[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:12 dist-test-slave-6lr6 krb5kdc[19381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:12 dist-test-slave-6lr6 krb5kdc[19381](info): set up 2 sockets
Sep 27 18:42:12 dist-test-slave-6lr6 krb5kdc[19381](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 18:42:13.121320 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:13.228400 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44181
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:44181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:13.329986 19394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:13.330201 19394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:13.330237 19394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:13.334076 19394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:13.334121 19394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:13.334138 19394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:13.334153 19394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:13.334172 19394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:13.339080 19394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:44181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:13.339951 19394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:13.340862 19394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:13.348806 19399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:13.348883 19402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:13.348866 19394 server_base.cc:1034] running on GCE node
W20260927 18:42:13.348812 19400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:13.349545 19394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:13.350522 19394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:13.351673 19394 hybrid_clock.cc:648] HybridClock initialized: now 1790534533351659 us; error 24 us; skew 500 ppm
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:13.354122 19394 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:13.354851 19394 webserver.cc:460] Webserver started at http://127.18.235.190:42777/ using document root <none> and password file <none>
I20260927 18:42:13.355309 19394 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:13.355373 19394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:13.355530 19394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:13.357388 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "c1b43fb5264848269e68a8d735217b8f"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "6bea09f24a73ae345666a3203681135e"
server_key_iv: "ebcd301f3adfb0afc7be28b29d9ddbcf"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.357882 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "c1b43fb5264848269e68a8d735217b8f"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "6bea09f24a73ae345666a3203681135e"
server_key_iv: "ebcd301f3adfb0afc7be28b29d9ddbcf"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.361068 19394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:42:13.363183 19409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.364239 19394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:13.364329 19394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "c1b43fb5264848269e68a8d735217b8f"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "6bea09f24a73ae345666a3203681135e"
server_key_iv: "ebcd301f3adfb0afc7be28b29d9ddbcf"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.364403 19394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:13.390664 19394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:13.393465 19394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:13.393600 19394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:13.399669 19394 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:44181
I20260927 18:42:13.399726 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:44181 every 8 connection(s)
I20260927 18:42:13.400516 19394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:13.403074 19462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:13.403324 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 19394
I20260927 18:42:13.403439 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:13.404958 19374 external_mini_cluster.cc:1428] Setting key 41c023d86059841e7c4c890a1cab3974
I20260927 18:42:13.408658 19462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f: Bootstrap starting.
I20260927 18:42:13.410707 19462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:13.411371 19462 log.cc:824] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:13.413013 19462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f: No bootstrap required, opened a new log
I20260927 18:42:13.416224 19462 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b43fb5264848269e68a8d735217b8f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44181 } }
I20260927 18:42:13.416430 19462 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:13.416486 19462 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b43fb5264848269e68a8d735217b8f, State: Initialized, Role: FOLLOWER
I20260927 18:42:13.416885 19462 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b43fb5264848269e68a8d735217b8f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44181 } }
I20260927 18:42:13.417006 19462 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:13.417068 19462 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:13.417176 19462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:13.417924 19462 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b43fb5264848269e68a8d735217b8f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44181 } }
I20260927 18:42:13.418215 19462 leader_election.cc:302] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1b43fb5264848269e68a8d735217b8f; no voters:
I20260927 18:42:13.418462 19462 leader_election.cc:288] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:13.418550 19467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:13.418757 19467 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [term 1 LEADER]: Becoming Leader. State: Replica: c1b43fb5264848269e68a8d735217b8f, State: Running, Role: LEADER
I20260927 18:42:13.419054 19467 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b43fb5264848269e68a8d735217b8f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44181 } }
I20260927 18:42:13.419571 19462 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:13.421386 19468 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1b43fb5264848269e68a8d735217b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b43fb5264848269e68a8d735217b8f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44181 } } }
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:13.421507 19468 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:13.421790 19469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b43fb5264848269e68a8d735217b8f. Latest consensus state: current_term: 1 leader_uuid: "c1b43fb5264848269e68a8d735217b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b43fb5264848269e68a8d735217b8f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44181 } } }
I20260927 18:42:13.422418 19469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f [sys.catalog]: This master's current role is: LEADER
W20260927 18:42:13.424434 19481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:13.424499 19481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:13.424580 19483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:13.425380 19483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:13.427958 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:13.407847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.18.235.190:44181)
0927 18:42:13.408271 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:13.408289 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:13.409585 (+ 1296us) server_negotiation.cc:407] Connection header received
0927 18:42:13.412499 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:13.412513 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:13.412809 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:13.413091 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:13.415135 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.416074 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.417339 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.417780 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.417922 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:13.422422 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:13.422428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:13.422442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:13.422469 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:13.424355 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:13.424983 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:13.424989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:13.424994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:13.425152 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:13.425522 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:13.425525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:13.425527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:13.426010 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:13.426191 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:13.426818 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 18:42:13.427093 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260927 18:42:13.430555 19483 catalog_manager.cc:1347] Generated new cluster ID: e7eb196c2ad4469ab82a8173c18a2cf6
I20260927 18:42:13.430601 19483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:13.444766 19483 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:13.445725 19483 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:13.452870 19483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1b43fb5264848269e68a8d735217b8f: Generated new TSK 0
I20260927 18:42:13.453588 19483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:13.483536 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44181
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:13.586459 19488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:13.586685 19488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:13.586735 19488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:13.590586 19488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:13.590634 19488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:13.590711 19488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:13.595681 19488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:13.596531 19488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:13.597467 19488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:13.605906 19494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:13.606017 19493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:13.605918 19496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:13.606060 19488 server_base.cc:1034] running on GCE node
I20260927 18:42:13.606491 19488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:13.607033 19488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:13.608172 19488 hybrid_clock.cc:648] HybridClock initialized: now 1790534533608155 us; error 28 us; skew 500 ppm
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:13.610522 19488 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:13.611246 19488 webserver.cc:460] Webserver started at http://127.18.235.129:33455/ using document root <none> and password file <none>
I20260927 18:42:13.611711 19488 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:13.611766 19488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:13.611944 19488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:13.613780 19488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "a508f014ea4c424597cfefa61c07cef4"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "a69ca635468eccdccf4d4a96b2d9cce4"
server_key_iv: "ad40f2a29c64cb7adc87155d41e2f4dc"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.614270 19488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "a508f014ea4c424597cfefa61c07cef4"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "a69ca635468eccdccf4d4a96b2d9cce4"
server_key_iv: "ad40f2a29c64cb7adc87155d41e2f4dc"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.617489 19488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:13.619513 19503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.620647 19488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:13.620784 19488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "a508f014ea4c424597cfefa61c07cef4"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "a69ca635468eccdccf4d4a96b2d9cce4"
server_key_iv: "ad40f2a29c64cb7adc87155d41e2f4dc"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.620882 19488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:13.644749 19488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:13.647325 19488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:13.647459 19488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:13.647909 19488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:13.648613 19488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:13.648655 19488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.648689 19488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:13.648710 19488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.656437 19488 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:35453
I20260927 18:42:13.656456 19616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:35453 every 8 connection(s)
I20260927 18:42:13.657385 19488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:13.658211 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 19488
I20260927 18:42:13.658288 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:13.658511 19374 external_mini_cluster.cc:1428] Setting key 8cb68c1f6ca4e6f6e56760bc98f3e6ce
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:42:13.670037 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:13.659143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:51933 (local address 127.18.235.190:44181)
0927 18:42:13.659263 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:13.659267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:13.660151 (+ 884us) server_negotiation.cc:407] Connection header received
0927 18:42:13.660962 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:13.660965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:13.661013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:13.661081 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:13.662192 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.662703 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.664241 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.664469 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.664539 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:13.666973 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:13.666977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:13.666979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:13.667003 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:13.668209 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:13.668656 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:13.668659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:13.668661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:13.668734 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:13.669009 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:13.669012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:13.669014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:13.669230 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:13.669325 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:13.669769 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 18:42:13.669883 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:13.670759 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:13.659394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44181 (local address 127.18.235.129:51933)
0927 18:42:13.660029 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:13.660065 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:13.660782 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:13.661269 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:13.661276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:13.661591 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:13.662054 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:13.662072 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.662808 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.662812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:13.664136 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:13.664148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.664597 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.664601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:13.664814 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:13.665366 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:13.665386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:13.666842 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:13.668312 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:13.668317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:13.668329 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:13.668570 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:13.668808 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:13.668811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:13.668812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:13.668929 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:13.669349 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:13.669354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:13.669553 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 18:42:13.669703 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:42:13.669916 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":106,"threads_started":1}
I20260927 18:42:13.671918 19617 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44181
I20260927 18:42:13.672134 19617 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:13.672631 19617 heartbeater.cc:511] Master 127.18.235.190:44181 requested a full tablet report, sending...
I20260927 18:42:13.674041 19426 ts_manager.cc:194] Registered new tserver with Master: a508f014ea4c424597cfefa61c07cef4 (127.18.235.129:35453)
I20260927 18:42:13.675091 19426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:51933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:13.683143 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44181
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:13.784083 19622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:13.784303 19622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:13.784343 19622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:13.788177 19622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:13.788223 19622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:13.788277 19622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:13.793079 19622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:13.793928 19622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:13.794770 19622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:13.803511 19628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:13.803474 19627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:13.803560 19622 server_base.cc:1034] running on GCE node
W20260927 18:42:13.803483 19630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:13.804044 19622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:13.804630 19622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:13.805797 19622 hybrid_clock.cc:648] HybridClock initialized: now 1790534533805780 us; error 27 us; skew 500 ppm
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:13.808244 19622 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:13.809031 19622 webserver.cc:460] Webserver started at http://127.18.235.130:37189/ using document root <none> and password file <none>
I20260927 18:42:13.809537 19622 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:13.809613 19622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:13.809808 19622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:13.811698 19622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "d2d6bc1c7273498fa174c0a78c3b2ab7"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "60e158b223b9074cf9b9ee320fb1384d"
server_key_iv: "6f83aa269a2b93e60a75720d5992adb5"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.812233 19622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "d2d6bc1c7273498fa174c0a78c3b2ab7"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "60e158b223b9074cf9b9ee320fb1384d"
server_key_iv: "6f83aa269a2b93e60a75720d5992adb5"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.815728 19622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:42:13.817994 19637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.819145 19622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:13.819244 19622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "d2d6bc1c7273498fa174c0a78c3b2ab7"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "60e158b223b9074cf9b9ee320fb1384d"
server_key_iv: "6f83aa269a2b93e60a75720d5992adb5"
server_key_version: "encryptionkey@0"
I20260927 18:42:13.819336 19622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:13.829607 19622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:13.831982 19622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:13.832146 19622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:13.832659 19622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:13.833418 19622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:13.833475 19622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.833528 19622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:13.833557 19622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:13.842442 19622 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:33925
I20260927 18:42:13.842466 19750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:33925 every 8 connection(s)
I20260927 18:42:13.843217 19622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:13.847422 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 19622
I20260927 18:42:13.847527 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:13.847826 19374 external_mini_cluster.cc:1428] Setting key 4acb729809932d66d393c418259b1267
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:13.856223 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:13.844722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:46607 (local address 127.18.235.190:44181)
0927 18:42:13.844873 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:13.844878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:13.845816 (+ 938us) server_negotiation.cc:407] Connection header received
0927 18:42:13.846557 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:13.846559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:13.846606 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:13.846679 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:13.847832 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.848410 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.850335 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.850554 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.850648 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:13.852976 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:13.852979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:13.852981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:13.853002 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:13.854226 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:13.854806 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:13.854808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:13.854809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:13.854872 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:13.855225 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:13.855227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:13.855228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:13.855388 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:13.855456 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:13.856026 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 18:42:13.856108 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:42:13.857309 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:13.845082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44181 (local address 127.18.235.130:46607)
0927 18:42:13.845683 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:13.845719 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:13.846399 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:13.846816 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:13.846822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:13.847118 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:13.847693 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:13.847714 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.848976 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.848980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:13.850222 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:13.850232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:13.850684 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:13.850688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:13.850872 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:13.851372 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:13.851387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:13.852817 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:13.854338 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:13.854344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:13.854359 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:13.854703 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:13.854951 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:13.854954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:13.854956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:13.855127 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:13.855484 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:13.855491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:13.855785 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 18:42:13.856010 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 18:42:13.856317 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":109,"threads_started":1}
I20260927 18:42:13.858312 19751 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44181
I20260927 18:42:13.858505 19751 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:13.858908 19751 heartbeater.cc:511] Master 127.18.235.190:44181 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:42:13.859841 19426 ts_manager.cc:194] Registered new tserver with Master: d2d6bc1c7273498fa174c0a78c3b2ab7 (127.18.235.130:33925)
I20260927 18:42:13.860353 19426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:46607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:13.872953 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44181
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:13.975384 19756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:13.975602 19756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:13.975647 19756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:13.979643 19756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:13.979701 19756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:13.979779 19756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:13.984732 19756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44181
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:13.985636 19756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:13.986521 19756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:13.995052 19762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:13.995217 19761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:13.995087 19764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:13.995323 19756 server_base.cc:1034] running on GCE node
I20260927 18:42:13.995747 19756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:13.996270 19756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:13.997408 19756 hybrid_clock.cc:648] HybridClock initialized: now 1790534533997386 us; error 38 us; skew 500 ppm
Sep 27 18:42:13 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:14.000234 19756 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:14.000952 19756 webserver.cc:460] Webserver started at http://127.18.235.131:36045/ using document root <none> and password file <none>
I20260927 18:42:14.001470 19756 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:14.001533 19756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:14.001709 19756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:14.003528 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "53db8681f7e24c55b02bb884620c3156"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "568349926866e024fd5dc6b0e39cbd6f"
server_key_iv: "1a114a351b181ed57af91fc2d3ee110e"
server_key_version: "encryptionkey@0"
I20260927 18:42:14.004026 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "53db8681f7e24c55b02bb884620c3156"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "568349926866e024fd5dc6b0e39cbd6f"
server_key_iv: "1a114a351b181ed57af91fc2d3ee110e"
server_key_version: "encryptionkey@0"
I20260927 18:42:14.007359 19756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:14.009521 19771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:14.010666 19756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:14.010768 19756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "53db8681f7e24c55b02bb884620c3156"
format_stamp: "Formatted at 2026-09-27 18:42:13 on dist-test-slave-6lr6"
server_key: "568349926866e024fd5dc6b0e39cbd6f"
server_key_iv: "1a114a351b181ed57af91fc2d3ee110e"
server_key_version: "encryptionkey@0"
I20260927 18:42:14.010862 19756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:14.023267 19756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:14.025869 19756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:14.026015 19756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:14.026490 19756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:14.027192 19756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:14.027235 19756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:14.027279 19756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:14.027307 19756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:14.035131 19756 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:39817
I20260927 18:42:14.035172 19884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:39817 every 8 connection(s)
I20260927 18:42:14.035914 19756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:14.037626 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 19756
I20260927 18:42:14.037716 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:14.037949 19374 external_mini_cluster.cc:1428] Setting key 7ca963b8424cca0ed777ec9ac9b69745
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534533, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:14.048053 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:14.037534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:52485 (local address 127.18.235.190:44181)
0927 18:42:14.037702 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:14.037708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:14.038808 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 18:42:14.039692 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:14.039696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:14.039752 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:14.039834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:14.040923 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.041510 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.042626 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.042860 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.042955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:14.045232 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:14.045236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:14.045238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:14.045259 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:14.046507 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:14.046926 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:14.046928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:14.046929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:14.046992 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:14.047248 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:14.047250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:14.047251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:14.047404 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:14.047469 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:14.047860 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:42:14.047943 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:42:14.048775 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:14.037938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44181 (local address 127.18.235.131:52485)
0927 18:42:14.038660 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:14.038711 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:14.039489 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:14.039988 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:14.039995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:14.040332 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:14.040780 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:14.040793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.041636 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.041641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:14.042496 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:14.042508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.042990 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.042994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:14.043171 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:14.043628 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:14.043645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:14.045033 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:14.046599 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:14.046605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:14.046614 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:14.046856 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:14.047054 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:14.047056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:14.047057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:14.047184 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:14.047475 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:14.047479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:14.047682 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:42:14.047832 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:42:14.048046 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":134,"threads_started":1}
I20260927 18:42:14.049727 19885 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44181
I20260927 18:42:14.049932 19885 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:14.050360 19885 heartbeater.cc:511] Master 127.18.235.190:44181 requested a full tablet report, sending...
I20260927 18:42:14.051251 19426 ts_manager.cc:194] Registered new tserver with Master: 53db8681f7e24c55b02bb884620c3156 (127.18.235.131:39817)
I20260927 18:42:14.051767 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:14.051796 19426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:52485
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:14.076071 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:14.067159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.18.235.129:35453)
0927 18:42:14.067472 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:14.067486 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:14.067545 (+ 59us) server_negotiation.cc:407] Connection header received
0927 18:42:14.067728 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:14.067734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:14.067906 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:14.068035 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:14.068779 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.069589 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.070431 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.070788 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.070893 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:14.073172 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:14.073180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:14.073191 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:14.073214 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:14.074446 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:14.074834 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:14.074840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:14.074842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:14.074935 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:14.075167 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:14.075170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:14.075172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:14.075406 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:14.075582 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:14.075785 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 18:42:14.075925 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
W20260927 18:42:14.076901 19531 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53812
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:14.098780 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:14.091476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.18.235.129:35453)
0927 18:42:14.091581 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:14.091585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:14.091737 (+ 152us) server_negotiation.cc:407] Connection header received
0927 18:42:14.091849 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:14.091852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:14.091888 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:14.091949 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:14.092538 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.093014 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.093800 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:14.094016 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:14.094100 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:14.096159 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:14.096162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:14.096164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:14.096184 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:14.097427 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:14.097770 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:14.097773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:14.097775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:14.097844 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:14.098067 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:14.098070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:14.098072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:14.098212 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:14.098329 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:14.098532 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 18:42:14.098637 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:42:14.099618 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 19488
I20260927 18:42:14.104254 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 19622
I20260927 18:42:14.108421 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 19756
I20260927 18:42:14.112432 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 19394
2026-09-27T18:42:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1951 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19899](info): set up 2 sockets
Sep 27 18:42:14 dist-test-slave-6lr6 krb5kdc[19899](info): commencing operation
krb5kdc: starting...
W20260927 18:42:15.048434 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:15.152894 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:36711
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:15.252700 19912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:15.252923 19912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:15.252974 19912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:15.256788 19912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:15.256839 19912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:15.256866 19912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:15.256886 19912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:15.256911 19912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:15.261739 19912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:36711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:15.262547 19912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:15.263381 19912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:15.270792 19918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.270787 19917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.270787 19920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:15.270816 19912 server_base.cc:1034] running on GCE node
I20260927 18:42:15.271488 19912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:15.272382 19912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:15.273548 19912 hybrid_clock.cc:648] HybridClock initialized: now 1790534535273532 us; error 30 us; skew 500 ppm
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:15.279266 19912 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:15.280002 19912 webserver.cc:460] Webserver started at http://127.18.235.190:35835/ using document root <none> and password file <none>
I20260927 18:42:15.280464 19912 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:15.280521 19912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:15.280685 19912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:15.282534 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "60ce32c0b262407b8216dd8b5f451f37"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "3c2c64ef0f12615a30ce8443540445d3"
server_key_iv: "45ed0c2cc5449f84024aedf3815813b8"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.283026 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "60ce32c0b262407b8216dd8b5f451f37"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "3c2c64ef0f12615a30ce8443540445d3"
server_key_iv: "45ed0c2cc5449f84024aedf3815813b8"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.286263 19912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:15.288331 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.289444 19912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:15.289544 19912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "60ce32c0b262407b8216dd8b5f451f37"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "3c2c64ef0f12615a30ce8443540445d3"
server_key_iv: "45ed0c2cc5449f84024aedf3815813b8"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.289628 19912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:15.300904 19912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:15.303481 19912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:15.303622 19912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:15.309473 19912 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:36711
I20260927 18:42:15.309507 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:36711 every 8 connection(s)
I20260927 18:42:15.310261 19912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:15.312595 19980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:15.317258 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 19912
I20260927 18:42:15.317353 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:15.317473 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37: Bootstrap starting.
I20260927 18:42:15.317561 19374 external_mini_cluster.cc:1428] Setting key 16064ec525384b701ae4ae697e2e6ff9
I20260927 18:42:15.319649 19980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:15.320297 19980 log.cc:824] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:15.321939 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37: No bootstrap required, opened a new log
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:15.325179 19980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce32c0b262407b8216dd8b5f451f37" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 36711 } }
I20260927 18:42:15.325378 19980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:15.325428 19980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ce32c0b262407b8216dd8b5f451f37, State: Initialized, Role: FOLLOWER
I20260927 18:42:15.325819 19980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce32c0b262407b8216dd8b5f451f37" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 36711 } }
I20260927 18:42:15.325937 19980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:15.325999 19980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:15.326090 19980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:15.327023 19980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce32c0b262407b8216dd8b5f451f37" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 36711 } }
I20260927 18:42:15.327351 19980 leader_election.cc:302] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ce32c0b262407b8216dd8b5f451f37; no voters:
I20260927 18:42:15.327646 19980 leader_election.cc:288] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:15.327739 19985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:15.327929 19985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [term 1 LEADER]: Becoming Leader. State: Replica: 60ce32c0b262407b8216dd8b5f451f37, State: Running, Role: LEADER
I20260927 18:42:15.328234 19985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce32c0b262407b8216dd8b5f451f37" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 36711 } }
I20260927 18:42:15.328733 19980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:15.329695 19986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ce32c0b262407b8216dd8b5f451f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce32c0b262407b8216dd8b5f451f37" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 36711 } } }
I20260927 18:42:15.329804 19986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:15.330147 19994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:15.330492 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.318701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.18.235.190:36711)
0927 18:42:15.319064 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.319077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.319108 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:42:15.319670 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.319685 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.320005 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.320277 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:15.320976 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.321947 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.322742 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.323183 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.323317 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.325659 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:15.325664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:15.325675 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:15.325712 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:15.327453 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.327914 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.327920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.327925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.328069 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.328423 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.328426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.328428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.328875 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:15.329079 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.329349 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.329630 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
I20260927 18:42:15.330983 19987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ce32c0b262407b8216dd8b5f451f37. Latest consensus state: current_term: 1 leader_uuid: "60ce32c0b262407b8216dd8b5f451f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ce32c0b262407b8216dd8b5f451f37" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 36711 } } }
I20260927 18:42:15.331111 19987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:15.333438 19994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:15.338084 19994 catalog_manager.cc:1347] Generated new cluster ID: d390b2e2c4644716b6d663fcf750ce70
I20260927 18:42:15.338176 19994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:15.348528 19994 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:15.349334 19994 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:15.352938 19994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ce32c0b262407b8216dd8b5f451f37: Generated new TSK 0
I20260927 18:42:15.353521 19994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:15.382098 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:36711
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:15.483376 20006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:15.483583 20006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:15.483637 20006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:15.487498 20006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:15.487545 20006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:15.487619 20006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:15.492404 20006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:36711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:15.493291 20006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:15.494160 20006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:15.502496 20011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.502556 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.502573 20014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:15.502662 20006 server_base.cc:1034] running on GCE node
I20260927 18:42:15.503082 20006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:15.503608 20006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:15.504752 20006 hybrid_clock.cc:648] HybridClock initialized: now 1790534535504732 us; error 30 us; skew 500 ppm
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:15.507082 20006 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:15.507814 20006 webserver.cc:460] Webserver started at http://127.18.235.129:36317/ using document root <none> and password file <none>
I20260927 18:42:15.508272 20006 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:15.508332 20006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:15.508519 20006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:15.510821 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "409d54fb83874c2a8e00bc1077cfc083"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "e3d2289b6591ab55cf77a1d0195f4115"
server_key_iv: "c38d504390047c6385366c26a7fd048d"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.511420 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "409d54fb83874c2a8e00bc1077cfc083"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "e3d2289b6591ab55cf77a1d0195f4115"
server_key_iv: "c38d504390047c6385366c26a7fd048d"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.514587 20006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:15.516577 20021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.517627 20006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:15.517730 20006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "409d54fb83874c2a8e00bc1077cfc083"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "e3d2289b6591ab55cf77a1d0195f4115"
server_key_iv: "c38d504390047c6385366c26a7fd048d"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.517815 20006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:15.533923 20006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:15.536331 20006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:15.536484 20006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:15.536998 20006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:15.537776 20006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:15.537819 20006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.537859 20006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:15.537889 20006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.545593 20006 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:43235
I20260927 18:42:15.545622 20134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:43235 every 8 connection(s)
I20260927 18:42:15.546346 20006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:15.546687 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20006
I20260927 18:42:15.546790 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:15.547066 19374 external_mini_cluster.cc:1428] Setting key c9f802b14fbb817fe55d8bfa33756b3f
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:42:15.558447 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.548056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:49273 (local address 127.18.235.190:36711)
0927 18:42:15.548176 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.548180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.549055 (+ 875us) server_negotiation.cc:407] Connection header received
0927 18:42:15.550108 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.550111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.550164 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.550241 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:15.551249 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.552031 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.553082 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.553466 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.553574 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.555618 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:15.555622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:15.555625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:15.555653 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:15.556771 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.557193 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.557196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.557198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.557269 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.557522 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.557524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.557525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.557717 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:15.557807 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.558243 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.558336 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:42:15.560176 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.548436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:36711 (local address 127.18.235.129:49273)
0927 18:42:15.548891 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:15.548940 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:15.549880 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:15.550384 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:15.550390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:15.550690 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:15.551115 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.551127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.552139 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.552141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.552965 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.552973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.553587 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.553590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.553717 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.554185 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:15.554200 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:15.555469 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:15.556866 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:15.556871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:15.556881 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:15.557098 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:15.557343 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:15.557345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:15.557346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:15.557459 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:15.557798 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:15.557802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:15.558014 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 18:42:15.558184 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 18:42:15.558514 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260927 18:42:15.561396 20135 heartbeater.cc:347] Connected to a master server at 127.18.235.190:36711
I20260927 18:42:15.561584 20135 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:15.561977 20135 heartbeater.cc:511] Master 127.18.235.190:36711 requested a full tablet report, sending...
I20260927 18:42:15.563265 19944 ts_manager.cc:194] Registered new tserver with Master: 409d54fb83874c2a8e00bc1077cfc083 (127.18.235.129:43235)
I20260927 18:42:15.564280 19944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:49273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:15.569985 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:36711
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:15.668982 20140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:15.669214 20140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:15.669263 20140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:15.673072 20140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:15.673141 20140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:15.673223 20140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:15.678462 20140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:36711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:15.679342 20140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:15.680202 20140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:15.688458 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:15.688517 20140 server_base.cc:1034] running on GCE node
W20260927 18:42:15.688436 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.688438 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:15.688973 20140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:15.689539 20140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:15.690692 20140 hybrid_clock.cc:648] HybridClock initialized: now 1790534535690681 us; error 31 us; skew 500 ppm
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:15.693236 20140 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:15.693982 20140 webserver.cc:460] Webserver started at http://127.18.235.130:33135/ using document root <none> and password file <none>
I20260927 18:42:15.694418 20140 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:15.694468 20140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:15.694612 20140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:15.696349 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "48bd69dd87c74b16a2df56cfdc89d538"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "1b2171cf528942a3b1123c4a39d3816d"
server_key_iv: "6d48e417bb35241e6148cc0b1b554a90"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.696861 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "48bd69dd87c74b16a2df56cfdc89d538"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "1b2171cf528942a3b1123c4a39d3816d"
server_key_iv: "6d48e417bb35241e6148cc0b1b554a90"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.700057 20140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:15.702135 20155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.703192 20140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:15.703276 20140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "48bd69dd87c74b16a2df56cfdc89d538"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "1b2171cf528942a3b1123c4a39d3816d"
server_key_iv: "6d48e417bb35241e6148cc0b1b554a90"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.703347 20140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:15.726833 20140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:15.729597 20140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:15.729732 20140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:15.730193 20140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:15.730899 20140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:15.730943 20140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.730978 20140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:15.730997 20140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.738725 20140 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:38843
I20260927 18:42:15.738752 20268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:38843 every 8 connection(s)
I20260927 18:42:15.739494 20140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:15.744264 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20140
I20260927 18:42:15.744369 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:15.744614 19374 external_mini_cluster.cc:1428] Setting key 310b5be578a368899b38166013f9ab47
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:15.751959 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.741028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:50629 (local address 127.18.235.190:36711)
0927 18:42:15.741169 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.741173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.741920 (+ 747us) server_negotiation.cc:407] Connection header received
0927 18:42:15.742734 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.742737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.742792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.742860 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:15.743894 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.744567 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.745881 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.746103 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.746190 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.748783 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:15.748786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:15.748788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:15.748812 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:15.750041 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.750568 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.750571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.750572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.750656 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.750983 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.750985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.750987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.751152 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:15.751224 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.751763 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.751850 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:42:15.752947 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.741394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:36711 (local address 127.18.235.130:50629)
0927 18:42:15.741788 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:15.741824 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:15.742540 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:15.743018 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:15.743025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:15.743336 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:15.743755 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.743767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.744693 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.744697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.745768 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.745778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.746239 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.746244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.746424 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.746903 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:15.746922 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:15.748606 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:15.750143 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:15.750149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:15.750162 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:15.750479 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:15.750733 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:15.750736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:15.750738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:15.750900 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:15.751245 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:15.751251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:15.751531 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 18:42:15.751741 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 18:42:15.752032 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:42:15.753988 20269 heartbeater.cc:347] Connected to a master server at 127.18.235.190:36711
I20260927 18:42:15.754186 20269 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:15.754608 20269 heartbeater.cc:511] Master 127.18.235.190:36711 requested a full tablet report, sending...
I20260927 18:42:15.755496 19944 ts_manager.cc:194] Registered new tserver with Master: 48bd69dd87c74b16a2df56cfdc89d538 (127.18.235.130:38843)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:42:15.756026 19944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:50629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:15.767623 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:36711
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:15.870535 20274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:15.870760 20274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:15.870815 20274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:15.874753 20274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:15.874801 20274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:15.874887 20274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:15.879825 20274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:36711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:15.880684 20274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:15.881623 20274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:15.890008 20280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.890002 20279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:15.890152 20282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:15.890168 20274 server_base.cc:1034] running on GCE node
I20260927 18:42:15.890542 20274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:15.891059 20274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:15.892200 20274 hybrid_clock.cc:648] HybridClock initialized: now 1790534535892178 us; error 38 us; skew 500 ppm
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:15.894713 20274 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:15.895435 20274 webserver.cc:460] Webserver started at http://127.18.235.131:45457/ using document root <none> and password file <none>
I20260927 18:42:15.895898 20274 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:15.895956 20274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:15.896117 20274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:15.897907 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "b56be77983834c548ffb5734b3599e7f"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "d19153e9269f438f0ba9076bf5360e68"
server_key_iv: "2c922ae9ec42bc922feceb4c21f71ede"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.898382 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "b56be77983834c548ffb5734b3599e7f"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "d19153e9269f438f0ba9076bf5360e68"
server_key_iv: "2c922ae9ec42bc922feceb4c21f71ede"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.901512 20274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:15.903568 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.904575 20274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:15.904667 20274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "b56be77983834c548ffb5734b3599e7f"
format_stamp: "Formatted at 2026-09-27 18:42:15 on dist-test-slave-6lr6"
server_key: "d19153e9269f438f0ba9076bf5360e68"
server_key_iv: "2c922ae9ec42bc922feceb4c21f71ede"
server_key_version: "encryptionkey@0"
I20260927 18:42:15.904747 20274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:15.922536 20274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:15.925096 20274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:15.925271 20274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:15.925736 20274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:15.926409 20274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:15.926451 20274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.926498 20274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:15.926523 20274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:15.934204 20274 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:36795
I20260927 18:42:15.934258 20402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:36795 every 8 connection(s)
I20260927 18:42:15.934944 20274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:15.942225 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20274
I20260927 18:42:15.942312 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:15.942521 19374 external_mini_cluster.cc:1428] Setting key fbbb79c30cb569a521832d41df1c2442
Sep 27 18:42:15 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:15.946007 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.936433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:35091 (local address 127.18.235.190:36711)
0927 18:42:15.936554 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.936558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.937280 (+ 722us) server_negotiation.cc:407] Connection header received
0927 18:42:15.938059 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.938062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.938116 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.938187 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:15.939190 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.939705 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.940751 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.940961 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.941044 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.943130 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:15.943133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:15.943135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:15.943161 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:15.944508 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.944903 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.944905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.944907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.944969 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.945239 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.945241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.945242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.945391 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:15.945458 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.945822 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.945900 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:42:15.946694 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.936723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:36711 (local address 127.18.235.131:35091)
0927 18:42:15.937142 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:15.937188 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:15.937878 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:15.938351 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:15.938357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:15.938650 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:15.939054 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.939066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.939813 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.939816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.940635 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.940644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.941082 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.941085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.941259 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.941661 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:15.941675 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:15.943003 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:15.944596 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:15.944600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:15.944609 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:15.944835 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:15.945025 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:15.945027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:15.945028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:15.945175 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:15.945459 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:15.945463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:15.945650 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 18:42:15.945778 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:42:15.945991 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260927 18:42:15.947613 20403 heartbeater.cc:347] Connected to a master server at 127.18.235.190:36711
I20260927 18:42:15.947789 20403 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:15.948181 20403 heartbeater.cc:511] Master 127.18.235.190:36711 requested a full tablet report, sending...
I20260927 18:42:15.948978 19944 ts_manager.cc:194] Registered new tserver with Master: b56be77983834c548ffb5734b3599e7f (127.18.235.131:36795)
I20260927 18:42:15.949509 19944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:35091
I20260927 18:42:15.955762 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:15.964950 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.958701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.18.235.190:36711)
0927 18:42:15.958792 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.958797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.958989 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:42:15.959098 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.959101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.959139 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.959198 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:15.959803 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.960273 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.961041 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.961302 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.961393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.962513 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:15.962516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:15.962519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:15.962541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:15.963559 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.963915 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.963918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.963920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.963986 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:15.964208 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:15.964211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:15.964213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:15.964406 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:15.964514 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.964722 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.964816 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:42:15.968787 19944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:15.971482 19944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:15.984181 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.979405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.18.235.131:36795)
0927 18:42:15.979793 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.979803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.979823 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:42:15.979889 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.979894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.980624 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.980771 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:15.980974 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.981887 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.982933 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.983542 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.983646 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.983763 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.983956 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.984049 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":111,"threads_started":1}
I20260927 18:42:15.984176 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.978663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38843 (local address 127.0.0.1:58408)
0927 18:42:15.978951 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:15.979000 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:15.979108 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:15.980042 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:15.980048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:15.980068 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:15.980193 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.980201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.981339 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.981343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.982650 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.982660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.983743 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.983746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.983839 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.983862 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:15.983910 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:42:15.984049 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":56,"thread_start_us":60,"threads_started":1}
I20260927 18:42:15.984189 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.979064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:36795 (local address 127.0.0.1:38978)
0927 18:42:15.979510 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:15.979524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:15.979604 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:15.980765 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:15.980767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:15.980785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:15.980869 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.980875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.981999 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.982002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.982815 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.982824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.983653 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.983657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.983773 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.983808 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:42:15.983909 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:42:15.984054 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":8,"thread_start_us":81,"threads_started":1}
I20260927 18:42:15.984540 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.979294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43235 (local address 127.0.0.1:48054)
0927 18:42:15.979555 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:15.979567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:15.979652 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:15.980227 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:15.980230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:15.980245 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:15.980337 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.980343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.981604 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.981608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.982727 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:15.982734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.983833 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.983837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:15.983912 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.983931 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:42:15.983987 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:42:15.984436 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":405,"thread_start_us":70,"threads_started":1}
I20260927 18:42:15.984846 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.979485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.18.235.129:43235)
0927 18:42:15.979820 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.979830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.979847 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:42:15.979906 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.979910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.980095 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.980254 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:15.980409 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.981478 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.982847 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.983595 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.983708 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.983833 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.984063 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.984152 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260927 18:42:15.984898 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:15.978841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.18.235.130:38843)
0927 18:42:15.979201 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:15.979215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:15.979238 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:42:15.979324 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:15.979329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:15.979895 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:15.980042 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:15.980315 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.981231 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.982783 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:15.983595 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:15.984547 (+ 952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:15.984654 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:15.984696 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 18:42:15.984770 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20260927 18:42:15.987040 20337 tablet_service.cc:1467] Processing CreateTablet for tablet e0dbd7aec2a042b9874d8cf61e7d7772 (DEFAULT_TABLE table=test-table [id=96205ce21e854130bd2e6c3b4883b945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:15.987290 20203 tablet_service.cc:1467] Processing CreateTablet for tablet e0dbd7aec2a042b9874d8cf61e7d7772 (DEFAULT_TABLE table=test-table [id=96205ce21e854130bd2e6c3b4883b945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:15.987291 20069 tablet_service.cc:1467] Processing CreateTablet for tablet e0dbd7aec2a042b9874d8cf61e7d7772 (DEFAULT_TABLE table=test-table [id=96205ce21e854130bd2e6c3b4883b945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:15.987937 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0dbd7aec2a042b9874d8cf61e7d7772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:15.988251 20203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0dbd7aec2a042b9874d8cf61e7d7772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:15.988251 20069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0dbd7aec2a042b9874d8cf61e7d7772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:15.993675 20420 tablet_bootstrap.cc:492] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: Bootstrap starting.
I20260927 18:42:15.995249 20422 tablet_bootstrap.cc:492] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538: Bootstrap starting.
I20260927 18:42:15.996227 20420 tablet_bootstrap.cc:654] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:15.997157 20420 log.cc:824] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:15.997220 20422 tablet_bootstrap.cc:654] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:15.997434 20421 tablet_bootstrap.cc:492] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: Bootstrap starting.
I20260927 18:42:15.997989 20422 log.cc:824] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:15.998798 20420 tablet_bootstrap.cc:492] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: No bootstrap required, opened a new log
I20260927 18:42:15.998963 20420 ts_tablet_manager.cc:1397] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:42:15.999562 20421 tablet_bootstrap.cc:654] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:15.999776 20422 tablet_bootstrap.cc:492] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538: No bootstrap required, opened a new log
I20260927 18:42:15.999943 20422 ts_tablet_manager.cc:1397] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 18:42:16.000319 20421 log.cc:824] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:16.002068 20421 tablet_bootstrap.cc:492] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: No bootstrap required, opened a new log
I20260927 18:42:16.002020 20420 raft_consensus.cc:348] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.002234 20421 ts_tablet_manager.cc:1397] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:16.002249 20420 raft_consensus.cc:374] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:16.002296 20420 raft_consensus.cc:729] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56be77983834c548ffb5734b3599e7f, State: Initialized, Role: FOLLOWER
I20260927 18:42:16.002673 20420 consensus_queue.cc:260] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.003206 20403 heartbeater.cc:503] Master 127.18.235.190:36711 was elected leader, sending a full tablet report...
I20260927 18:42:16.003521 20420 ts_tablet_manager.cc:1428] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:16.004281 20422 raft_consensus.cc:348] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.004525 20422 raft_consensus.cc:374] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:16.004575 20422 raft_consensus.cc:729] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bd69dd87c74b16a2df56cfdc89d538, State: Initialized, Role: FOLLOWER
I20260927 18:42:16.004990 20422 consensus_queue.cc:260] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.005882 20422 ts_tablet_manager.cc:1428] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 18:42:16.005882 20421 raft_consensus.cc:348] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.006111 20421 raft_consensus.cc:374] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:16.006157 20421 raft_consensus.cc:729] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409d54fb83874c2a8e00bc1077cfc083, State: Initialized, Role: FOLLOWER
I20260927 18:42:16.006186 20269 heartbeater.cc:503] Master 127.18.235.190:36711 was elected leader, sending a full tablet report...
I20260927 18:42:16.006556 20421 consensus_queue.cc:260] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.007121 20135 heartbeater.cc:503] Master 127.18.235.190:36711 was elected leader, sending a full tablet report...
I20260927 18:42:16.008270 20421 ts_tablet_manager.cc:1428] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 18:42:16.047329 20136 tablet.cc:2369] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:16.122601 20428 raft_consensus.cc:482] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:16.122761 20428 raft_consensus.cc:504] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.123749 20428 leader_election.cc:288] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48bd69dd87c74b16a2df56cfdc89d538 (127.18.235.130:38843), b56be77983834c548ffb5734b3599e7f (127.18.235.131:36795)
I20260927 18:42:16.127477 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.124137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:47193 (local address 127.18.235.130:38843)
0927 18:42:16.124263 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:16.124267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:16.124346 (+ 79us) server_negotiation.cc:407] Connection header received
0927 18:42:16.124555 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:16.124558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:16.124611 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:16.124684 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:16.124925 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.125495 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.126526 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.126923 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.127019 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.127069 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:16.127282 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 18:42:16.127354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:42:16.127483 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.124099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:57711 (local address 127.18.235.131:36795)
0927 18:42:16.124217 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:16.124221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:16.124469 (+ 248us) server_negotiation.cc:407] Connection header received
0927 18:42:16.124619 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:16.124622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:16.124673 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:16.124748 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:16.125054 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.125540 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.126550 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.126931 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.127029 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.127076 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:16.127299 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:42:16.127364 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:42:16.127516 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.123950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:36795 (local address 127.18.235.129:57711)
0927 18:42:16.124378 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:16.124389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:16.124485 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:16.124810 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:16.124813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:16.124829 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:16.124950 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:16.124956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.125639 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.125641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:16.126442 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:16.126450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.127031 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.127035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:16.127192 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.127210 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:42:16.127276 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:42:16.127364 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":128,"thread_start_us":65,"threads_started":1}
I20260927 18:42:16.127525 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.123950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38843 (local address 127.18.235.129:47193)
0927 18:42:16.124274 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:16.124287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:16.124418 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:16.124692 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:16.124695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:16.124711 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:16.124833 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:16.124840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.125594 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.125598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:16.126418 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:16.126427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.127027 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.127030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:16.127174 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.127199 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:42:16.127254 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 18:42:16.127359 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":76,"thread_start_us":94,"threads_started":1}
I20260927 18:42:16.128235 20223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0dbd7aec2a042b9874d8cf61e7d7772" candidate_uuid: "409d54fb83874c2a8e00bc1077cfc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48bd69dd87c74b16a2df56cfdc89d538" is_pre_election: true
I20260927 18:42:16.128234 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0dbd7aec2a042b9874d8cf61e7d7772" candidate_uuid: "409d54fb83874c2a8e00bc1077cfc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56be77983834c548ffb5734b3599e7f" is_pre_election: true
I20260927 18:42:16.128476 20223 raft_consensus.cc:2454] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409d54fb83874c2a8e00bc1077cfc083 in term 0.
I20260927 18:42:16.128484 20357 raft_consensus.cc:2454] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409d54fb83874c2a8e00bc1077cfc083 in term 0.
I20260927 18:42:16.128868 20022 leader_election.cc:302] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409d54fb83874c2a8e00bc1077cfc083, 48bd69dd87c74b16a2df56cfdc89d538; no voters:
I20260927 18:42:16.129313 20428 raft_consensus.cc:2781] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:16.129379 20428 raft_consensus.cc:482] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:16.129420 20428 raft_consensus.cc:3037] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:16.130442 20428 raft_consensus.cc:504] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.130730 20428 leader_election.cc:288] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [CANDIDATE]: Term 1 election: Requested vote from peers 48bd69dd87c74b16a2df56cfdc89d538 (127.18.235.130:38843), b56be77983834c548ffb5734b3599e7f (127.18.235.131:36795)
I20260927 18:42:16.131037 20223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0dbd7aec2a042b9874d8cf61e7d7772" candidate_uuid: "409d54fb83874c2a8e00bc1077cfc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48bd69dd87c74b16a2df56cfdc89d538"
I20260927 18:42:16.131073 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0dbd7aec2a042b9874d8cf61e7d7772" candidate_uuid: "409d54fb83874c2a8e00bc1077cfc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b56be77983834c548ffb5734b3599e7f"
I20260927 18:42:16.131134 20223 raft_consensus.cc:3037] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:16.131170 20357 raft_consensus.cc:3037] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:16.132082 20223 raft_consensus.cc:2454] T e0dbd7aec2a042b9874d8cf61e7d7772 P 48bd69dd87c74b16a2df56cfdc89d538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409d54fb83874c2a8e00bc1077cfc083 in term 1.
I20260927 18:42:16.132231 20357 raft_consensus.cc:2454] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409d54fb83874c2a8e00bc1077cfc083 in term 1.
I20260927 18:42:16.132332 20022 leader_election.cc:302] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409d54fb83874c2a8e00bc1077cfc083, 48bd69dd87c74b16a2df56cfdc89d538; no voters:
I20260927 18:42:16.132488 20428 raft_consensus.cc:2781] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:16.132711 20428 raft_consensus.cc:686] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [term 1 LEADER]: Becoming Leader. State: Replica: 409d54fb83874c2a8e00bc1077cfc083, State: Running, Role: LEADER
I20260927 18:42:16.132989 20428 consensus_queue.cc:237] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } }
I20260927 18:42:16.135812 19944 catalog_manager.cc:5495] T e0dbd7aec2a042b9874d8cf61e7d7772 P 409d54fb83874c2a8e00bc1077cfc083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 409d54fb83874c2a8e00bc1077cfc083 (127.18.235.129). New cstate: current_term: 1 leader_uuid: "409d54fb83874c2a8e00bc1077cfc083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd69dd87c74b16a2df56cfdc89d538" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "409d54fb83874c2a8e00bc1077cfc083" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b56be77983834c548ffb5734b3599e7f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36795 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:16.170149 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.162441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.18.235.129:43235)
0927 18:42:16.162588 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:16.162593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:16.162728 (+ 135us) server_negotiation.cc:407] Connection header received
0927 18:42:16.162776 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:16.162778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:16.162822 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:16.162897 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:16.163470 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.163980 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.164750 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.164958 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.165057 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.167224 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:16.167235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:16.167247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:16.167270 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:16.168670 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:16.169026 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:16.169031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:16.169033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:16.169151 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:16.169353 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:16.169355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:16.169356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:16.169584 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:16.169785 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:16.169865 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:42:16.169956 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534536, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 18:42:16.185791 20404 tablet.cc:2369] T e0dbd7aec2a042b9874d8cf61e7d7772 P b56be77983834c548ffb5734b3599e7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534536, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:16.193563 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.186333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.18.235.129:43235)
0927 18:42:16.186452 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:16.186457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:16.186583 (+ 126us) server_negotiation.cc:407] Connection header received
0927 18:42:16.186705 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:16.186708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:16.186747 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:16.186806 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:16.187396 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.187879 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.188627 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.188858 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.188942 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.190983 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:16.190986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:16.190988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:16.191010 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:16.192041 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:16.192409 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:16.192413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:16.192415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:16.192484 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:16.192713 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:16.192717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:16.192719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:16.192875 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:16.193012 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:16.193302 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:42:16.193421 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[19899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[19899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:16.215356 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:16.208597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.18.235.129:43235)
0927 18:42:16.208710 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:16.208715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:16.208837 (+ 122us) server_negotiation.cc:407] Connection header received
0927 18:42:16.208936 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:16.208939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:16.208979 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:16.209038 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:16.209658 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.210141 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.210899 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:16.211108 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:16.211187 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:16.213194 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:16.213197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:16.213198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:16.213218 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:16.214202 (+ 984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:16.214514 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:16.214516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:16.214518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:16.214575 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:16.214756 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:16.214758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:16.214759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:16.214897 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:16.214996 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:16.215123 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:42:16.215222 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 18:42:16.216526 20049 tablet_service.cc:2908] Rejecting scan request for tablet e0dbd7aec2a042b9874d8cf61e7d7772: Uninitialized: clean time has not yet been initialized
I20260927 18:42:16.218209 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20006
I20260927 18:42:16.223136 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20140
I20260927 18:42:16.227422 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20274
I20260927 18:42:16.231653 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 19912
2026-09-27T18:42:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2119 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[20445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[20445](info): set up 2 sockets
Sep 27 18:42:16 dist-test-slave-6lr6 krb5kdc[20445](info): commencing operation
krb5kdc: starting...
W20260927 18:42:17.168412 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:17.273445 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38879
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:17.373095 20458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:17.373332 20458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:17.373368 20458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:17.377183 20458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:17.377223 20458 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:42:17.377238 20458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:17.377255 20458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:17.377272 20458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:17.377287 20458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:17.382158 20458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:38879
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:17.382956 20458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:17.383782 20458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:17.391202 20464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:17.391196 20463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:17.391252 20458 server_base.cc:1034] running on GCE node
W20260927 18:42:17.391207 20466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:17.391911 20458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:17.392822 20458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:17.393971 20458 hybrid_clock.cc:648] HybridClock initialized: now 1790534537393954 us; error 28 us; skew 500 ppm
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:17.396288 20458 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:17.397051 20458 webserver.cc:460] Webserver started at http://127.18.235.190:34917/ using document root <none> and password file <none>
I20260927 18:42:17.397545 20458 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:17.397599 20458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:17.397758 20458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:17.399495 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "8e7903b5deec4cee943bd054653bd377"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "f92d73656cf6af98b7a1fd5dc2299eb3"
server_key_iv: "3cbdddaa5462e110d5583046ea017bb7"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.399971 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "8e7903b5deec4cee943bd054653bd377"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "f92d73656cf6af98b7a1fd5dc2299eb3"
server_key_iv: "3cbdddaa5462e110d5583046ea017bb7"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.403157 20458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:17.405174 20473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:17.406284 20458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:17.406380 20458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8e7903b5deec4cee943bd054653bd377"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "f92d73656cf6af98b7a1fd5dc2299eb3"
server_key_iv: "3cbdddaa5462e110d5583046ea017bb7"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.406454 20458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:17.429426 20458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:17.431751 20458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:17.431878 20458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:17.438784 20458 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:38879
I20260927 18:42:17.438798 20535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:38879 every 8 connection(s)
I20260927 18:42:17.439584 20458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:17.442071 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:17.446890 20536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: Bootstrap starting.
I20260927 18:42:17.447887 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20458
I20260927 18:42:17.447973 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:17.448189 19374 external_mini_cluster.cc:1428] Setting key d307594f46dc85b29d8bd777e803b499
I20260927 18:42:17.449245 20536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:17.450045 20536 log.cc:824] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:17.451964 20536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: No bootstrap required, opened a new log
I20260927 18:42:17.455693 20536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7903b5deec4cee943bd054653bd377" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38879 } }
I20260927 18:42:17.455897 20536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:17.455960 20536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7903b5deec4cee943bd054653bd377, State: Initialized, Role: FOLLOWER
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:17.456339 20536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7903b5deec4cee943bd054653bd377" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38879 } }
I20260927 18:42:17.456470 20536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:17.456530 20536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:17.456615 20536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:17.457545 20536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7903b5deec4cee943bd054653bd377" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38879 } }
I20260927 18:42:17.457877 20536 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7903b5deec4cee943bd054653bd377; no voters:
I20260927 18:42:17.458189 20536 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:17.458274 20541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:17.458455 20541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7903b5deec4cee943bd054653bd377, State: Running, Role: LEADER
I20260927 18:42:17.458765 20541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7903b5deec4cee943bd054653bd377" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38879 } }
I20260927 18:42:17.459285 20536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:17.460314 20543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7903b5deec4cee943bd054653bd377. Latest consensus state: current_term: 1 leader_uuid: "8e7903b5deec4cee943bd054653bd377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7903b5deec4cee943bd054653bd377" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38879 } } }
I20260927 18:42:17.460314 20542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7903b5deec4cee943bd054653bd377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7903b5deec4cee943bd054653bd377" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38879 } } }
I20260927 18:42:17.460443 20543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:17.460443 20542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:17.460767 20550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:17.462248 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.449123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.18.235.190:38879)
0927 18:42:17.449541 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:17.449559 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:17.449596 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:42:17.450353 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:17.450377 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:17.450680 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:17.451006 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:17.451629 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.453009 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.453825 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.454496 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.454651 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.456916 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:17.456921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:17.456936 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:17.456962 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:17.458703 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.459145 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.459150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.459155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.459281 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.459627 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.459631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.459633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.460099 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:17.460922 (+ 823us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:17.461080 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:42:17.461380 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260927 18:42:17.464581 20550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:17.469501 20550 catalog_manager.cc:1347] Generated new cluster ID: bfad0a3ad095411a9ed14b19cadfe396
I20260927 18:42:17.469545 20550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:17.477166 20550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:17.477936 20550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:17.481809 20550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: Generated new TSK 0
I20260927 18:42:17.482432 20550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:17.513865 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38879
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:17.615589 20562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:17.615803 20562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:17.615849 20562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:17.619717 20562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:17.619761 20562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:17.619807 20562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:17.619858 20562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:17.624655 20562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:38879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:17.625506 20562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:17.626366 20562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:17.634725 20568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:17.634845 20567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:17.634691 20570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:17.634907 20562 server_base.cc:1034] running on GCE node
I20260927 18:42:17.635326 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:17.635866 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:17.637012 20562 hybrid_clock.cc:648] HybridClock initialized: now 1790534537636997 us; error 25 us; skew 500 ppm
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:17.639433 20562 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:17.640154 20562 webserver.cc:460] Webserver started at http://127.18.235.129:38605/ using document root <none> and password file <none>
I20260927 18:42:17.640636 20562 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:17.640694 20562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:17.640858 20562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:17.642704 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "81288136ac214cfaa3b92b3f1b18ea59"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "3cd37211104c8c7ba1247f51ec218031"
server_key_iv: "046a537fc4617a0a149593c0adefa92d"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.643196 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "81288136ac214cfaa3b92b3f1b18ea59"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "3cd37211104c8c7ba1247f51ec218031"
server_key_iv: "046a537fc4617a0a149593c0adefa92d"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.646548 20562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:17.648608 20577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:17.649669 20562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:17.649763 20562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "81288136ac214cfaa3b92b3f1b18ea59"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "3cd37211104c8c7ba1247f51ec218031"
server_key_iv: "046a537fc4617a0a149593c0adefa92d"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.649843 20562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:17.658910 20562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:17.661626 20562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:17.661777 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:17.662947 20562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:17.662997 20562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:17.663043 20562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:17.663071 20562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:17.672200 20562 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:40583
I20260927 18:42:17.672472 20692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:40583 every 8 connection(s)
I20260927 18:42:17.672993 20562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:17.674696 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.663608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:52463 (local address 127.18.235.190:38879)
0927 18:42:17.663757 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:17.663762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:17.664445 (+ 683us) server_negotiation.cc:407] Connection header received
0927 18:42:17.665469 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:17.665472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:17.665527 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:17.665608 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:17.666995 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.667777 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.669035 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.669338 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.669425 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.671523 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:17.671532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:17.671535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:17.671559 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:17.672843 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.673342 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.673345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.673346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.673430 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.673763 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.673766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.673767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.673938 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:17.674012 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:17.674496 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 18:42:17.674584 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:42:17.675448 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.663904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.18.235.129:52463)
0927 18:42:17.664299 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:17.664343 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:17.665223 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:17.665829 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:17.665840 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:17.666249 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:17.666854 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.666873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.667886 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.667890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.668929 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.668939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.669473 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.669477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.669631 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.670071 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:17.670085 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:17.671381 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:17.672941 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.672946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.672958 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.673243 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.673521 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.673524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.673526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.673676 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.674020 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:17.674024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:17.674296 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 18:42:17.674463 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 18:42:17.674702 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260927 18:42:17.676558 20693 heartbeater.cc:347] Connected to a master server at 127.18.235.190:38879
I20260927 18:42:17.676682 20693 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:17.677151 20693 heartbeater.cc:511] Master 127.18.235.190:38879 requested a full tablet report, sending...
I20260927 18:42:17.678535 20489 ts_manager.cc:194] Registered new tserver with Master: 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129:40583)
I20260927 18:42:17.679100 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20562
I20260927 18:42:17.679214 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:17.679468 19374 external_mini_cluster.cc:1428] Setting key 16f9583b3a66a6518b0e557bc60baa1b
I20260927 18:42:17.679943 20489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:52463
I20260927 18:42:17.684871 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.678289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:32985 (local address 127.18.235.190:38879)
0927 18:42:17.678397 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:17.678402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:17.678562 (+ 160us) server_negotiation.cc:407] Connection header received
0927 18:42:17.678668 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:17.678671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:17.678709 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:17.678767 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:17.679160 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.679777 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.681178 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.681383 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.681460 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.682356 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:17.682359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:17.682360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:17.682379 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:17.683537 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.683887 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.683891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.683892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.683954 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.684176 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.684179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.684181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.684335 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:17.684416 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:17.684639 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:42:17.684732 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":43}
I20260927 18:42:17.684877 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.678226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.18.235.129:32985)
0927 18:42:17.678495 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:17.678504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:17.678583 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:17.678792 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:17.678795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:17.678971 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:17.679054 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.679058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.679899 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.679902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.680644 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.680653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.681484 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.681487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.681581 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.681953 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:17.681966 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:17.682232 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:17.683627 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.683631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.683633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.683800 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.684027 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.684030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.684032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.684104 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.684419 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:17.684422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:17.684503 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:42:17.684582 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:42:17.684697 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:17.703557 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38879
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:17.805188 20702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:17.805388 20702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:17.805434 20702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:17.809327 20702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:17.809369 20702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:17.809412 20702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:17.809463 20702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:17.814288 20702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:38879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:17.815086 20702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:17.815937 20702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:17.824193 20710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:17.824236 20707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:17.824246 20702 server_base.cc:1034] running on GCE node
W20260927 18:42:17.824330 20708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:17.824759 20702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:17.825311 20702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:17.826459 20702 hybrid_clock.cc:648] HybridClock initialized: now 1790534537826442 us; error 28 us; skew 500 ppm
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:17.828760 20702 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:17.829545 20702 webserver.cc:460] Webserver started at http://127.18.235.130:46241/ using document root <none> and password file <none>
I20260927 18:42:17.830011 20702 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:17.830070 20702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:17.830245 20702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:17.832002 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "884b0ece648c471da7201d112a0cdd1e"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "b8102c20fa5d6d84106de0d6d29b8857"
server_key_iv: "9943dae94cc89b68ea1c1182ce04c60b"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.832495 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "884b0ece648c471da7201d112a0cdd1e"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "b8102c20fa5d6d84106de0d6d29b8857"
server_key_iv: "9943dae94cc89b68ea1c1182ce04c60b"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.835770 20702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:17.837913 20717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:17.838955 20702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:17.839051 20702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "884b0ece648c471da7201d112a0cdd1e"
format_stamp: "Formatted at 2026-09-27 18:42:17 on dist-test-slave-6lr6"
server_key: "b8102c20fa5d6d84106de0d6d29b8857"
server_key_iv: "9943dae94cc89b68ea1c1182ce04c60b"
server_key_version: "encryptionkey@0"
I20260927 18:42:17.839133 20702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:17.856756 20702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:17.859119 20702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:17.859277 20702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:17.860594 20702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:17.860639 20702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:17.860687 20702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:17.860715 20702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:17 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:17.870174 20702 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:38817
I20260927 18:42:17.870220 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:38817 every 8 connection(s)
I20260927 18:42:17.870906 20702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:17.872766 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.861194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:55983 (local address 127.18.235.190:38879)
0927 18:42:17.861299 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:17.861304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:17.862293 (+ 989us) server_negotiation.cc:407] Connection header received
0927 18:42:17.863147 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:17.863151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:17.863209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:17.863298 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:17.864432 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.865245 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.866439 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.866660 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.866743 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.869698 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:17.869701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:17.869702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:17.869726 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:17.871127 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.871670 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.871674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.871676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.871750 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.872004 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.872006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.872007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.872157 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:17.872222 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:17.872572 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:42:17.872653 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:42:17.873507 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.861746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.18.235.130:55983)
0927 18:42:17.862167 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:17.862202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:17.862951 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:17.863501 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:17.863509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:17.863887 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:17.864292 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.864304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.865363 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.865366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.866322 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.866331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.866796 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.866800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.866966 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.867437 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:17.867455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:17.869547 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:17.871245 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.871250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.871263 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.871571 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.871821 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.871823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.871825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.871940 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.872223 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:17.872227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:17.872410 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 18:42:17.872541 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:42:17.872757 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260927 18:42:17.874519 20833 heartbeater.cc:347] Connected to a master server at 127.18.235.190:38879
I20260927 18:42:17.874635 20833 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:17.875073 20833 heartbeater.cc:511] Master 127.18.235.190:38879 requested a full tablet report, sending...
I20260927 18:42:17.875957 20490 ts_manager.cc:194] Registered new tserver with Master: 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817)
I20260927 18:42:17.876643 20490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:55983
I20260927 18:42:17.878034 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20702
I20260927 18:42:17.878129 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:17.878364 19374 external_mini_cluster.cc:1428] Setting key 923a060ad07747ae3a47cafcf8b1a27d
I20260927 18:42:17.882475 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.876235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:57683 (local address 127.18.235.190:38879)
0927 18:42:17.876336 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:17.876340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:17.876535 (+ 195us) server_negotiation.cc:407] Connection header received
0927 18:42:17.876635 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:17.876637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:17.876671 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:17.876731 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:17.877121 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.877648 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.878632 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.878846 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.878927 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.879894 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:17.879898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:17.879900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:17.879921 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:17.881161 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.881552 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.881555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.881556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.881616 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:17.881845 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:17.881847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:17.881848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:17.882013 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:17.882080 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:17.882302 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 18:42:17.882379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:42:17.882547 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:17.876157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.18.235.130:57683)
0927 18:42:17.876472 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:17.876484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:17.876566 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:17.876759 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:17.876761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:17.876933 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:17.877019 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.877024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.877757 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.877760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.878532 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:17.878541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:17.878951 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:17.878954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:17.879021 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:17.879493 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:17.879503 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:17.879771 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:17.881252 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.881255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.881257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.881472 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.881688 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:17.881690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:17.881692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:17.881773 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:17.882092 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:17.882094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:17.882172 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 18:42:17.882265 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:42:17.882380 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:17.901228 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38879
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:18.002027 20842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:18.002234 20842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:18.002274 20842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:18.006028 20842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:18.006071 20842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:18.006106 20842 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:18.006146 20842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:18.011050 20842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:38879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:18.011870 20842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:18.012727 20842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:18.021906 20850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:18.021855 20847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:18.022025 20848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:18.022254 20842 server_base.cc:1034] running on GCE node
I20260927 18:42:18.022639 20842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:18.023221 20842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:18.024396 20842 hybrid_clock.cc:648] HybridClock initialized: now 1790534538024383 us; error 28 us; skew 500 ppm
Sep 27 18:42:18 dist-test-slave-6lr6 krb5kdc[20445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534538, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:18.026921 20842 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:18.027714 20842 webserver.cc:460] Webserver started at http://127.18.235.131:34087/ using document root <none> and password file <none>
I20260927 18:42:18.028211 20842 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:18.028278 20842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:18.028472 20842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:18.030416 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "0745aba2966147f395559856e2685cae"
format_stamp: "Formatted at 2026-09-27 18:42:18 on dist-test-slave-6lr6"
server_key: "8000fe46701c962874fe01bbdba77788"
server_key_iv: "a6964798d4cac73274f3a7eff33fdf62"
server_key_version: "encryptionkey@0"
I20260927 18:42:18.030939 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "0745aba2966147f395559856e2685cae"
format_stamp: "Formatted at 2026-09-27 18:42:18 on dist-test-slave-6lr6"
server_key: "8000fe46701c962874fe01bbdba77788"
server_key_iv: "a6964798d4cac73274f3a7eff33fdf62"
server_key_version: "encryptionkey@0"
I20260927 18:42:18.034386 20842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 18:42:18.036648 20857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:18.037760 20842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:18.037865 20842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "0745aba2966147f395559856e2685cae"
format_stamp: "Formatted at 2026-09-27 18:42:18 on dist-test-slave-6lr6"
server_key: "8000fe46701c962874fe01bbdba77788"
server_key_iv: "a6964798d4cac73274f3a7eff33fdf62"
server_key_version: "encryptionkey@0"
I20260927 18:42:18.037948 20842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:18.066371 20842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:18.068784 20842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:18.068940 20842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:18.070245 20842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:18.070290 20842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:18.070338 20842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:18.070364 20842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:18 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534538, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:18.079541 20842 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:40097
I20260927 18:42:18.079604 20972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:40097 every 8 connection(s)
I20260927 18:42:18.080344 20842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:18.082712 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.070947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:58913 (local address 127.18.235.190:38879)
0927 18:42:18.071096 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.071101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.072320 (+ 1219us) server_negotiation.cc:407] Connection header received
0927 18:42:18.073440 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.073443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.073497 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.073571 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:18.074767 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.075334 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.076625 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.076963 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.077052 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.079620 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:18.079625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:18.079628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:18.079653 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:18.081224 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.081617 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.081620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.081621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.081696 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.081928 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.081929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.081930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.082076 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:18.082140 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.082509 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.082587 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:42:18.083455 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.071681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.18.235.131:58913)
0927 18:42:18.072175 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.072220 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.073243 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.073761 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.073770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.074080 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:18.074631 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.074647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.075496 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.075500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.076519 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.076528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.077088 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.077093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.077271 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.077786 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:18.077806 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:18.079440 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:18.081316 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:18.081321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:18.081330 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:18.081548 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:18.081754 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:18.081757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:18.081758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:18.081870 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:18.082139 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:18.082143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:18.082341 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.082477 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.082682 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":110,"threads_started":1}
I20260927 18:42:18.084556 20973 heartbeater.cc:347] Connected to a master server at 127.18.235.190:38879
I20260927 18:42:18.084677 20973 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:18.085078 20973 heartbeater.cc:511] Master 127.18.235.190:38879 requested a full tablet report, sending...
I20260927 18:42:18.085525 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 20842
I20260927 18:42:18.085628 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:18.085894 19374 external_mini_cluster.cc:1428] Setting key aa2ad46c5a36bc025ed42b91f18d5da2
I20260927 18:42:18.086023 20490 ts_manager.cc:194] Registered new tserver with Master: 0745aba2966147f395559856e2685cae (127.18.235.131:40097)
I20260927 18:42:18.086692 20490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:58913
I20260927 18:42:18.088160 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:18.092707 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.086488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:34473 (local address 127.18.235.190:38879)
0927 18:42:18.086581 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.086585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.086742 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:42:18.086895 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.086898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.086942 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.087017 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:18.087453 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.087929 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.088977 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.089285 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.089376 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.090253 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:18.090256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:18.090257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:18.090276 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:18.091358 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.091733 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.091737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.091739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.091826 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.092043 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.092046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.092047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.092234 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:18.092338 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.092510 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.092608 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:42:18.092759 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.086440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.18.235.131:34473)
0927 18:42:18.086667 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.086681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.086814 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.087044 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.087047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.087238 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:18.087351 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.087357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.088158 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.088161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.088870 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.088878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.089374 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.089376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.089438 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.089846 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:18.089860 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:18.090133 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:18.091447 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:18.091451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:18.091453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:18.091629 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:18.091905 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:18.091907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:18.091909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:18.091973 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:18.092313 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:18.092314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:18.092376 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.092446 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.092563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
Sep 27 18:42:18 dist-test-slave-6lr6 krb5kdc[20445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:42:18 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:18.110296 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.103796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.18.235.190:38879)
0927 18:42:18.103897 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.103900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.103911 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:18.103945 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.103948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.103983 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.104044 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:18.104622 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.105127 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.105855 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.106044 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.106106 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.108062 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:18.108065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:18.108066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:18.108086 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:18.109062 (+ 976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.109414 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.109417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.109418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.109494 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.109669 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.109672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.109674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.109825 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:18.109916 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.110087 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.110165 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:42:18.112695 20490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:18.115550 20490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:18.128399 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.122820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:40097 (local address 127.0.0.1:39188)
0927 18:42:18.123533 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.123582 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.123688 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.124569 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.124572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.124593 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.124756 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.124763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.125972 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.125975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.127117 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.127126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.128008 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.128010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.128083 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.128109 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.128176 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.128262 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":61,"spinlock_wait_cycles":12032,"thread_start_us":70,"threads_started":1}
I20260927 18:42:18.128424 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.123529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.18.235.131:40097)
0927 18:42:18.124022 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.124033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.124054 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:42:18.124119 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.124125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.124326 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.124467 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.125175 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.125867 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.127221 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.127912 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.128043 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.128162 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.128209 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.128292 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":99,"threads_started":1}
I20260927 18:42:18.128700 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.122737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40583 (local address 127.0.0.1:48274)
0927 18:42:18.123522 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.123570 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.123700 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.124425 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.124431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.124453 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.124564 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.124571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.125940 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.125943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.127387 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.127394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.128335 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.128338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.128425 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.128447 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.128519 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.128580 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":147,"thread_start_us":82,"threads_started":1}
I20260927 18:42:18.129007 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.122900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.18.235.129:40583)
0927 18:42:18.123979 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.123989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.124011 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:18.124085 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.124090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.124287 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.124433 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.124675 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.125830 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.127486 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.128228 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.128363 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.128470 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.128810 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.128885 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":122,"threads_started":1}
I20260927 18:42:18.130445 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.123147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38817 (local address 127.0.0.1:34180)
0927 18:42:18.123887 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.123898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.123970 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.124569 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.124572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.124597 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.124761 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.124766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.126033 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.126036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.127368 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.127379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.129412 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.129416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.130173 (+ 757us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.130196 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.130263 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.130323 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":145,"spinlock_wait_cycles":3584,"thread_start_us":83,"threads_started":1}
I20260927 18:42:18.130915 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.123453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.18.235.130:38817)
0927 18:42:18.123801 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.123813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.124048 (+ 235us) server_negotiation.cc:407] Connection header received
0927 18:42:18.124109 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.124113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.124286 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.124433 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.124870 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.125945 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.128541 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.129298 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.129426 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.129536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.130674 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.130768 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":79,"threads_started":1}
I20260927 18:42:18.131238 20907 tablet_service.cc:1467] Processing CreateTablet for tablet a00a4965793841b4a3d42969cee48307 (DEFAULT_TABLE table=test-table [id=8e26213636bc471ba927cdf7fef499f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:18.131528 20627 tablet_service.cc:1467] Processing CreateTablet for tablet a00a4965793841b4a3d42969cee48307 (DEFAULT_TABLE table=test-table [id=8e26213636bc471ba927cdf7fef499f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:18.132241 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00a4965793841b4a3d42969cee48307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.132434 20627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00a4965793841b4a3d42969cee48307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.133507 20767 tablet_service.cc:1467] Processing CreateTablet for tablet a00a4965793841b4a3d42969cee48307 (DEFAULT_TABLE table=test-table [id=8e26213636bc471ba927cdf7fef499f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:18.134423 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00a4965793841b4a3d42969cee48307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.138911 20996 tablet_bootstrap.cc:492] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Bootstrap starting.
I20260927 18:42:18.140784 20996 tablet_bootstrap.cc:654] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.141422 20998 tablet_bootstrap.cc:492] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Bootstrap starting.
I20260927 18:42:18.141585 20996 log.cc:824] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:18.143209 20998 tablet_bootstrap.cc:654] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.143276 20996 tablet_bootstrap.cc:492] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: No bootstrap required, opened a new log
I20260927 18:42:18.143448 20996 ts_tablet_manager.cc:1397] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:42:18.143896 20998 log.cc:824] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:18.144431 20997 tablet_bootstrap.cc:492] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Bootstrap starting.
I20260927 18:42:18.145686 20998 tablet_bootstrap.cc:492] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: No bootstrap required, opened a new log
I20260927 18:42:18.145859 20998 ts_tablet_manager.cc:1397] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:42:18.146011 20997 tablet_bootstrap.cc:654] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.146600 20997 log.cc:824] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:18.147111 20996 raft_consensus.cc:348] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.147359 20996 raft_consensus.cc:374] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.147411 20996 raft_consensus.cc:729] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81288136ac214cfaa3b92b3f1b18ea59, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.147833 20996 consensus_queue.cc:260] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.148145 20997 tablet_bootstrap.cc:492] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: No bootstrap required, opened a new log
I20260927 18:42:18.148319 20997 ts_tablet_manager.cc:1397] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 18:42:18.148535 20693 heartbeater.cc:503] Master 127.18.235.190:38879 was elected leader, sending a full tablet report...
I20260927 18:42:18.148746 20996 ts_tablet_manager.cc:1428] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:42:18.149566 20998 raft_consensus.cc:348] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.149783 20998 raft_consensus.cc:374] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.149837 20998 raft_consensus.cc:729] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884b0ece648c471da7201d112a0cdd1e, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.150199 20998 consensus_queue.cc:260] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.151060 20998 ts_tablet_manager.cc:1428] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:18.151388 20833 heartbeater.cc:503] Master 127.18.235.190:38879 was elected leader, sending a full tablet report...
I20260927 18:42:18.151743 20997 raft_consensus.cc:348] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.151954 20997 raft_consensus.cc:374] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.151996 20997 raft_consensus.cc:729] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0745aba2966147f395559856e2685cae, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.152367 20997 consensus_queue.cc:260] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.153044 20973 heartbeater.cc:503] Master 127.18.235.190:38879 was elected leader, sending a full tablet report...
I20260927 18:42:18.153293 20997 ts_tablet_manager.cc:1428] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260927 18:42:18.174028 20694 tablet.cc:2369] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:18.331226 20974 tablet.cc:2369] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:18.372030 20834 tablet.cc:2369] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:18.485818 21002 raft_consensus.cc:482] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:18.486009 21002 raft_consensus.cc:504] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.487119 21002 leader_election.cc:288] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0745aba2966147f395559856e2685cae (127.18.235.131:40097), 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817)
I20260927 18:42:18.490835 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.487431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:34413 (local address 127.18.235.130:38817)
0927 18:42:18.487550 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.487555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.487763 (+ 208us) server_negotiation.cc:407] Connection header received
0927 18:42:18.487856 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.487858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.487895 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.487960 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.488219 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.488704 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.489864 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.490241 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.490337 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.490386 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.490634 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.490714 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:42:18.490839 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.487478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:40279 (local address 127.18.235.131:40097)
0927 18:42:18.487598 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.487602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.487691 (+ 89us) server_negotiation.cc:407] Connection header received
0927 18:42:18.487818 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.487822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.487870 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.487923 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.488190 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.488697 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.489867 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.490256 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.490360 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.490409 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.490634 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.490706 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:18.490867 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.487333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38817 (local address 127.18.235.129:34413)
0927 18:42:18.487666 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.487680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.487776 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.487977 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.487979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.488015 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.488116 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.488120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.488800 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.488803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.489746 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.489755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.490350 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.490353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.490499 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.490540 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.490602 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.490705 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":159,"spinlock_wait_cycles":768,"thread_start_us":66,"threads_started":1}
I20260927 18:42:18.490866 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.487322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:40097 (local address 127.18.235.129:40279)
0927 18:42:18.487612 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.487624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.487714 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.487954 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.487957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.487971 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.488090 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.488095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.488801 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.488804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.489749 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.489757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.490356 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.490360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.490514 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.490539 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.490602 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.490724 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":116,"spinlock_wait_cycles":3840,"thread_start_us":82,"threads_started":1}
I20260927 18:42:18.491448 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00a4965793841b4a3d42969cee48307" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884b0ece648c471da7201d112a0cdd1e" is_pre_election: true
I20260927 18:42:18.491535 20925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00a4965793841b4a3d42969cee48307" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0745aba2966147f395559856e2685cae" is_pre_election: true
I20260927 18:42:18.491717 20787 raft_consensus.cc:2454] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 0.
I20260927 18:42:18.491786 20925 raft_consensus.cc:2454] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 0.
I20260927 18:42:18.492131 20580 leader_election.cc:302] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81288136ac214cfaa3b92b3f1b18ea59, 884b0ece648c471da7201d112a0cdd1e; no voters:
I20260927 18:42:18.492360 21002 raft_consensus.cc:2781] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:18.492437 21002 raft_consensus.cc:482] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:18.492483 21002 raft_consensus.cc:3037] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.493441 21002 raft_consensus.cc:504] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.493709 21002 leader_election.cc:288] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 election: Requested vote from peers 0745aba2966147f395559856e2685cae (127.18.235.131:40097), 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817)
I20260927 18:42:18.493986 20925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00a4965793841b4a3d42969cee48307" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0745aba2966147f395559856e2685cae"
I20260927 18:42:18.494010 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00a4965793841b4a3d42969cee48307" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884b0ece648c471da7201d112a0cdd1e"
I20260927 18:42:18.494120 20787 raft_consensus.cc:3037] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.494120 20925 raft_consensus.cc:3037] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.495291 20787 raft_consensus.cc:2454] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 1.
I20260927 18:42:18.495579 20580 leader_election.cc:302] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81288136ac214cfaa3b92b3f1b18ea59, 884b0ece648c471da7201d112a0cdd1e; no voters:
I20260927 18:42:18.495715 20925 raft_consensus.cc:2454] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 1.
I20260927 18:42:18.495759 21002 raft_consensus.cc:2781] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:18.495908 21002 raft_consensus.cc:686] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 LEADER]: Becoming Leader. State: Replica: 81288136ac214cfaa3b92b3f1b18ea59, State: Running, Role: LEADER
I20260927 18:42:18.496194 21002 consensus_queue.cc:237] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } }
I20260927 18:42:18.499022 20490 catalog_manager.cc:5495] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129). New cstate: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:18.578109 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.574593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.18.235.129:40583)
0927 18:42:18.574715 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.574719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.574731 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:18.574899 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.574901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.574952 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.575030 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:18.575341 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.575913 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.576820 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.577042 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.577148 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.577417 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.577779 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.577860 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.577933 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.577978 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:42:18.583408 20787 raft_consensus.cc:1261] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Refusing update from remote peer 81288136ac214cfaa3b92b3f1b18ea59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:18.584057 21007 consensus_queue.cc:1035] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:18.584151 20925 raft_consensus.cc:1261] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Refusing update from remote peer 81288136ac214cfaa3b92b3f1b18ea59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:18.584620 21007 consensus_queue.cc:1035] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:18.591506 21012 mvcc.cc:205] Tried to move back new op lower bound from 7334029470030888960 to 7334029469682515968. Current Snapshot: MvccSnapshot[applied={T|T < 7334029470030888960}]
I20260927 18:42:18.592852 21011 mvcc.cc:205] Tried to move back new op lower bound from 7334029470030888960 to 7334029469682515968. Current Snapshot: MvccSnapshot[applied={T|T < 7334029470030888960}]
I20260927 18:42:18.600518 20490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:18.600664 20490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:18.602932 20490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: Sending DeleteTablet for 3 replicas of tablet a00a4965793841b4a3d42969cee48307
I20260927 18:42:18.603582 20627 tablet_service.cc:1514] Processing DeleteTablet for tablet a00a4965793841b4a3d42969cee48307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:18 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:48274
I20260927 18:42:18.603751 20767 tablet_service.cc:1514] Processing DeleteTablet for tablet a00a4965793841b4a3d42969cee48307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:18 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:34180
I20260927 18:42:18.603984 20907 tablet_service.cc:1514] Processing DeleteTablet for tablet a00a4965793841b4a3d42969cee48307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:18 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:39188
I20260927 18:42:18.604064 21019 tablet_replica.cc:330] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: stopping tablet replica
I20260927 18:42:18.604125 21020 tablet_replica.cc:330] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: stopping tablet replica
I20260927 18:42:18.604395 21020 raft_consensus.cc:2229] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:18.604399 21019 raft_consensus.cc:2229] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:18.604664 21020 raft_consensus.cc:2258] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:18.604965 21021 tablet_replica.cc:330] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: stopping tablet replica
I20260927 18:42:18.605050 20490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:42:18.605229 21021 raft_consensus.cc:2229] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:18.605458 21021 raft_consensus.cc:2258] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 18:42:18.605491 20490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:18.605947 21020 ts_tablet_manager.cc:1905] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:18.606288 21019 raft_consensus.cc:2258] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:18.607828 21021 ts_tablet_manager.cc:1905] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:18.608472 21020 ts_tablet_manager.cc:1918] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:18.608546 21020 log.cc:1196] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/a00a4965793841b4a3d42969cee48307
I20260927 18:42:18.608795 21020 ts_tablet_manager.cc:1939] T a00a4965793841b4a3d42969cee48307 P 884b0ece648c471da7201d112a0cdd1e: Deleting consensus metadata
I20260927 18:42:18.609944 20476 catalog_manager.cc:4841] TS 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817): tablet a00a4965793841b4a3d42969cee48307 (table test-table [id=8e26213636bc471ba927cdf7fef499f2]) successfully deleted
I20260927 18:42:18.610481 21021 ts_tablet_manager.cc:1918] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:18.610563 21021 log.cc:1196] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/a00a4965793841b4a3d42969cee48307
I20260927 18:42:18.610927 20627 tablet_service.cc:1467] Processing CreateTablet for tablet bf9c605134bb42109618eebe4d274cbd (DEFAULT_TABLE table=test-table [id=ec95fffa5f684311a40b43a58691e813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:18.611016 21021 ts_tablet_manager.cc:1939] T a00a4965793841b4a3d42969cee48307 P 0745aba2966147f395559856e2685cae: Deleting consensus metadata
I20260927 18:42:18.611158 20627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9c605134bb42109618eebe4d274cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.612205 21019 ts_tablet_manager.cc:1905] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:18.613507 20996 tablet_bootstrap.cc:492] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Bootstrap starting.
I20260927 18:42:18.614316 21019 ts_tablet_manager.cc:1918] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:18.614393 21019 log.cc:1196] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/a00a4965793841b4a3d42969cee48307
I20260927 18:42:18.614403 20996 tablet_bootstrap.cc:654] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.614619 21019 ts_tablet_manager.cc:1939] T a00a4965793841b4a3d42969cee48307 P 81288136ac214cfaa3b92b3f1b18ea59: Deleting consensus metadata
I20260927 18:42:18.615072 20477 catalog_manager.cc:4841] TS 0745aba2966147f395559856e2685cae (127.18.235.131:40097): tablet a00a4965793841b4a3d42969cee48307 (table test-table [id=8e26213636bc471ba927cdf7fef499f2]) successfully deleted
I20260927 18:42:18.615487 20767 tablet_service.cc:1467] Processing CreateTablet for tablet bf9c605134bb42109618eebe4d274cbd (DEFAULT_TABLE table=test-table [id=ec95fffa5f684311a40b43a58691e813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:18.615705 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9c605134bb42109618eebe4d274cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.617467 20475 catalog_manager.cc:4841] TS 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129:40583): tablet a00a4965793841b4a3d42969cee48307 (table test-table [id=8e26213636bc471ba927cdf7fef499f2]) successfully deleted
I20260927 18:42:18.617522 20907 tablet_service.cc:1467] Processing CreateTablet for tablet bf9c605134bb42109618eebe4d274cbd (DEFAULT_TABLE table=test-table [id=ec95fffa5f684311a40b43a58691e813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:18.617731 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9c605134bb42109618eebe4d274cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.617749 20998 tablet_bootstrap.cc:492] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Bootstrap starting.
I20260927 18:42:18.618000 20996 tablet_bootstrap.cc:492] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: No bootstrap required, opened a new log
I20260927 18:42:18.618069 20996 ts_tablet_manager.cc:1397] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:42:18.618463 20996 raft_consensus.cc:348] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.618556 20996 raft_consensus.cc:374] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.618587 20998 tablet_bootstrap.cc:654] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.618587 20996 raft_consensus.cc:729] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81288136ac214cfaa3b92b3f1b18ea59, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.618709 20996 consensus_queue.cc:260] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.618963 20996 ts_tablet_manager.cc:1428] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:18.619809 20997 tablet_bootstrap.cc:492] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Bootstrap starting.
I20260927 18:42:18.620548 20998 tablet_bootstrap.cc:492] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: No bootstrap required, opened a new log
I20260927 18:42:18.620620 20998 ts_tablet_manager.cc:1397] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:42:18.620641 20997 tablet_bootstrap.cc:654] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.621002 20998 raft_consensus.cc:348] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.621129 20998 raft_consensus.cc:374] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.621167 20998 raft_consensus.cc:729] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884b0ece648c471da7201d112a0cdd1e, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.621275 20998 consensus_queue.cc:260] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.621495 20998 ts_tablet_manager.cc:1428] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:18.621922 20997 tablet_bootstrap.cc:492] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: No bootstrap required, opened a new log
I20260927 18:42:18.621994 20997 ts_tablet_manager.cc:1397] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:18.622398 20997 raft_consensus.cc:348] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.622502 20997 raft_consensus.cc:374] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.622532 20997 raft_consensus.cc:729] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0745aba2966147f395559856e2685cae, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.622630 20997 consensus_queue.cc:260] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.622874 20997 ts_tablet_manager.cc:1428] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:18.670707 21004 raft_consensus.cc:482] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:18.670862 21004 raft_consensus.cc:504] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.672133 21004 leader_election.cc:288] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129:40583), 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817)
I20260927 18:42:18.677086 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.673051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:52825 (local address 127.18.235.129:40583)
0927 18:42:18.673202 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.673206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.673219 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:18.673263 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.673266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.673307 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.673370 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.673700 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.674408 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.675922 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.676450 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.676531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.676583 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.676909 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.676977 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:42:18.677626 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.672384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40583 (local address 127.18.235.131:52825)
0927 18:42:18.672791 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.672805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.672903 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.673471 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.673474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.673491 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.673598 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.673603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.674535 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.674537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.675790 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.675800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.676688 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.676692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.676784 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.676805 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.677415 (+ 610us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.677487 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":61,"threads_started":1}
I20260927 18:42:18.678049 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.672795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:36845 (local address 127.18.235.130:38817)
0927 18:42:18.672937 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.672942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.673325 (+ 383us) server_negotiation.cc:407] Connection header received
0927 18:42:18.673520 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.673523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.673571 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.673635 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:18.674044 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.674826 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.676661 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.677286 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.677398 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.677449 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.677845 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.677921 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:42:18.678216 20647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c605134bb42109618eebe4d274cbd" candidate_uuid: "0745aba2966147f395559856e2685cae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81288136ac214cfaa3b92b3f1b18ea59" is_pre_election: true
I20260927 18:42:18.678438 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.672452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38817 (local address 127.18.235.131:36845)
0927 18:42:18.673237 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.673251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.673399 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.673770 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.673774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.673790 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:18.673919 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.673925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.675185 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.675188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.676496 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.676508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.677575 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.677579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.677697 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.677722 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.678230 (+ 508us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.678300 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":60,"threads_started":1}
I20260927 18:42:18.678444 20647 raft_consensus.cc:2454] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0745aba2966147f395559856e2685cae in term 0.
I20260927 18:42:18.678822 20859 leader_election.cc:302] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0745aba2966147f395559856e2685cae, 81288136ac214cfaa3b92b3f1b18ea59; no voters:
I20260927 18:42:18.678839 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c605134bb42109618eebe4d274cbd" candidate_uuid: "0745aba2966147f395559856e2685cae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884b0ece648c471da7201d112a0cdd1e" is_pre_election: true
I20260927 18:42:18.678968 20787 raft_consensus.cc:2454] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0745aba2966147f395559856e2685cae in term 0.
I20260927 18:42:18.679136 21004 raft_consensus.cc:2781] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:18.679200 21004 raft_consensus.cc:482] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:18.679243 21004 raft_consensus.cc:3037] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.680444 21004 raft_consensus.cc:504] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.680820 21004 leader_election.cc:288] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [CANDIDATE]: Term 1 election: Requested vote from peers 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129:40583), 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817)
I20260927 18:42:18.681144 20647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c605134bb42109618eebe4d274cbd" candidate_uuid: "0745aba2966147f395559856e2685cae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81288136ac214cfaa3b92b3f1b18ea59"
I20260927 18:42:18.681262 20647 raft_consensus.cc:3037] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.681339 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9c605134bb42109618eebe4d274cbd" candidate_uuid: "0745aba2966147f395559856e2685cae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884b0ece648c471da7201d112a0cdd1e"
I20260927 18:42:18.681438 20787 raft_consensus.cc:3037] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.682340 20647 raft_consensus.cc:2454] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0745aba2966147f395559856e2685cae in term 1.
I20260927 18:42:18.682468 20787 raft_consensus.cc:2454] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0745aba2966147f395559856e2685cae in term 1.
I20260927 18:42:18.682636 20859 leader_election.cc:302] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0745aba2966147f395559856e2685cae, 81288136ac214cfaa3b92b3f1b18ea59; no voters:
I20260927 18:42:18.682788 21004 raft_consensus.cc:2781] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:18.682967 21004 raft_consensus.cc:686] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 1 LEADER]: Becoming Leader. State: Replica: 0745aba2966147f395559856e2685cae, State: Running, Role: LEADER
I20260927 18:42:18.683307 21004 consensus_queue.cc:237] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.685858 20490 catalog_manager.cc:5495] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae reported cstate change: term changed from 0 to 1, leader changed from <none> to 0745aba2966147f395559856e2685cae (127.18.235.131). New cstate: current_term: 1 leader_uuid: "0745aba2966147f395559856e2685cae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } health_report { overall_health: HEALTHY } } }
Sep 27 18:42:18 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534537, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:18.727574 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.718718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.18.235.190:38879)
0927 18:42:18.719068 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.719083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.719101 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:42:18.719158 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.719161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.719213 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.719331 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:18.720172 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.721055 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.721886 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.722255 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.722392 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.724690 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:18.724694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:18.724697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:18.724724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:18.726109 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.726503 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.726506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.726507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.726578 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:18.726804 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:18.726807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:18.726809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:18.726963 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:18.727042 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.727299 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.727438 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":269,"thread_start_us":80,"threads_started":1}
I20260927 18:42:18.727573 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.718612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38879 (local address 127.0.0.1:53900)
0927 18:42:18.718953 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.718964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.719048 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.719361 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.719364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.719891 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:18.720019 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.720027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.721214 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.721216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.721774 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.721781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.722374 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.722376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.722431 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.723000 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:18.723011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:18.724550 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:18.726204 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:18.726214 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:18.726220 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:18.726421 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:18.726651 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:18.726654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:18.726656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:18.726733 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:18.727053 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:18.727059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:18.727155 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.727245 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.727386 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":90,"threads_started":1}
I20260927 18:42:18.731034 20490 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:53900:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:42:18.731747 20490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:18.736120 20627 tablet_service.cc:1467] Processing CreateTablet for tablet 8f58e9da1ad6438c923e0120d5be22f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d6782427a2c48548476b89c57676355]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:18.736227 20907 tablet_service.cc:1467] Processing CreateTablet for tablet 8f58e9da1ad6438c923e0120d5be22f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d6782427a2c48548476b89c57676355]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:18.736327 20767 tablet_service.cc:1467] Processing CreateTablet for tablet 8f58e9da1ad6438c923e0120d5be22f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d6782427a2c48548476b89c57676355]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:18.736507 20627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f58e9da1ad6438c923e0120d5be22f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.736527 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f58e9da1ad6438c923e0120d5be22f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.736675 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f58e9da1ad6438c923e0120d5be22f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:18.739115 20996 tablet_bootstrap.cc:492] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: Bootstrap starting.
I20260927 18:42:18.739377 20997 tablet_bootstrap.cc:492] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: Bootstrap starting.
I20260927 18:42:18.740015 20998 tablet_bootstrap.cc:492] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: Bootstrap starting.
I20260927 18:42:18.740300 20997 tablet_bootstrap.cc:654] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.740338 20996 tablet_bootstrap.cc:654] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.740902 20998 tablet_bootstrap.cc:654] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:18.741772 20997 tablet_bootstrap.cc:492] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: No bootstrap required, opened a new log
I20260927 18:42:18.741866 20997 ts_tablet_manager.cc:1397] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 18:42:18.742007 20996 tablet_bootstrap.cc:492] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: No bootstrap required, opened a new log
I20260927 18:42:18.742093 20996 ts_tablet_manager.cc:1397] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:42:18.742321 20997 raft_consensus.cc:348] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.742424 20997 raft_consensus.cc:374] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.742455 20997 raft_consensus.cc:729] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0745aba2966147f395559856e2685cae, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.742506 20996 raft_consensus.cc:348] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.742599 20996 raft_consensus.cc:374] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.742559 20997 consensus_queue.cc:260] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.742628 20996 raft_consensus.cc:729] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81288136ac214cfaa3b92b3f1b18ea59, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.742748 20996 consensus_queue.cc:260] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.742810 20997 ts_tablet_manager.cc:1428] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:18.743018 21007 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.743044 21004 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.743149 21007 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.743149 21004 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.744042 20996 ts_tablet_manager.cc:1428] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:18.745381 20998 tablet_bootstrap.cc:492] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: No bootstrap required, opened a new log
I20260927 18:42:18.745466 20998 ts_tablet_manager.cc:1397] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:42:18.745847 20998 raft_consensus.cc:348] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.745939 20998 raft_consensus.cc:374] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:18.745965 20998 raft_consensus.cc:729] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884b0ece648c471da7201d112a0cdd1e, State: Initialized, Role: FOLLOWER
I20260927 18:42:18.746065 20998 consensus_queue.cc:260] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.746301 20998 ts_tablet_manager.cc:1428] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:18.746305 21003 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.746455 21003 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.911754 21007 raft_consensus.cc:482] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:18.911870 21007 raft_consensus.cc:504] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.912256 21007 leader_election.cc:288] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817), 0745aba2966147f395559856e2685cae (127.18.235.131:40097)
I20260927 18:42:18.912608 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f58e9da1ad6438c923e0120d5be22f2" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884b0ece648c471da7201d112a0cdd1e" is_pre_election: true
I20260927 18:42:18.912658 20925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f58e9da1ad6438c923e0120d5be22f2" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0745aba2966147f395559856e2685cae" is_pre_election: true
I20260927 18:42:18.912760 20787 raft_consensus.cc:2454] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 0.
I20260927 18:42:18.912776 20925 raft_consensus.cc:2454] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 0.
I20260927 18:42:18.913012 20581 leader_election.cc:302] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0745aba2966147f395559856e2685cae, 81288136ac214cfaa3b92b3f1b18ea59; no voters:
I20260927 18:42:18.913166 21007 raft_consensus.cc:2781] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:18.913223 21007 raft_consensus.cc:482] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:18.913265 21007 raft_consensus.cc:3037] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.914180 21007 raft_consensus.cc:504] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.914440 21007 leader_election.cc:288] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 election: Requested vote from peers 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817), 0745aba2966147f395559856e2685cae (127.18.235.131:40097)
I20260927 18:42:18.914752 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f58e9da1ad6438c923e0120d5be22f2" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884b0ece648c471da7201d112a0cdd1e"
I20260927 18:42:18.914750 20925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f58e9da1ad6438c923e0120d5be22f2" candidate_uuid: "81288136ac214cfaa3b92b3f1b18ea59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0745aba2966147f395559856e2685cae"
I20260927 18:42:18.914863 20787 raft_consensus.cc:3037] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.914863 20925 raft_consensus.cc:3037] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:18.915863 20925 raft_consensus.cc:2454] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 1.
I20260927 18:42:18.915863 20787 raft_consensus.cc:2454] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81288136ac214cfaa3b92b3f1b18ea59 in term 1.
I20260927 18:42:18.916096 20581 leader_election.cc:302] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0745aba2966147f395559856e2685cae, 81288136ac214cfaa3b92b3f1b18ea59; no voters:
I20260927 18:42:18.916255 21007 raft_consensus.cc:2781] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:18.916345 21007 raft_consensus.cc:686] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 LEADER]: Becoming Leader. State: Replica: 81288136ac214cfaa3b92b3f1b18ea59, State: Running, Role: LEADER
I20260927 18:42:18.916473 21007 consensus_queue.cc:237] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } }
I20260927 18:42:18.917063 21002 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: TxnStatusTablet state changed. Reason: New leader 81288136ac214cfaa3b92b3f1b18ea59. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.917176 21002 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:18.917426 21040 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:42:18.917599 21040 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:42:18.918074 20488 catalog_manager.cc:5495] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129). New cstate: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:18.931408 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.927830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40583 (local address 127.0.0.1:48294)
0927 18:42:18.927997 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.928012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.928113 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.928513 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.928516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.928524 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:18.928641 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.928647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.929664 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.929667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.930270 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.930279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.930818 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.930822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.930893 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.930936 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.931192 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.931200 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.931240 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.931310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:42:18.931427 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.927938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.18.235.129:40583)
0927 18:42:18.928296 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.928300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.928312 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:18.928351 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.928354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.928397 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.928465 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:18.928740 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.929537 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.930372 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.930700 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.930794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.931042 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.931116 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.931182 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.931262 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.931306 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260927 18:42:18.933431 20925 raft_consensus.cc:1261] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Refusing update from remote peer 81288136ac214cfaa3b92b3f1b18ea59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:18.933506 20787 raft_consensus.cc:1261] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Refusing update from remote peer 81288136ac214cfaa3b92b3f1b18ea59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:18.933830 21002 consensus_queue.cc:1035] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:18.933971 21007 consensus_queue.cc:1035] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:18.935187 21004 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: TxnStatusTablet state changed. Reason: New leader 81288136ac214cfaa3b92b3f1b18ea59. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.935216 21003 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: TxnStatusTablet state changed. Reason: New leader 81288136ac214cfaa3b92b3f1b18ea59. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.935293 21004 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.935308 21003 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.936642 21007 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.936739 21007 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:18.936820 21010 mvcc.cc:205] Tried to move back new op lower bound from 7334029471468060672 to 7334029471403040768. Current Snapshot: MvccSnapshot[applied={T|T < 7334029471468060672}]
I20260927 18:42:18.938652 21003 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.938738 21003 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 884b0ece648c471da7201d112a0cdd1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.940999 21002 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.941098 21002 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 81288136ac214cfaa3b92b3f1b18ea59: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:18.942027 21004 tablet_replica.cc:439] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81288136ac214cfaa3b92b3f1b18ea59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 } } peers { permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 } } peers { permanent_uuid: "0745aba2966147f395559856e2685cae" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40097 } } }
I20260927 18:42:18.942134 21004 tablet_replica.cc:442] T 8f58e9da1ad6438c923e0120d5be22f2 P 0745aba2966147f395559856e2685cae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:18.946441 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.942816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.18.235.131:40097)
0927 18:42:18.942911 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.942915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.942928 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:18.942992 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.942995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.943050 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.943144 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:18.943461 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.944212 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.944960 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.945324 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.945418 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.945602 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.946102 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.946222 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.946251 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.946303 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:42:18.955914 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.952570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:51645 (local address 127.18.235.129:40583)
0927 18:42:18.952680 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.952685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.952914 (+ 229us) server_negotiation.cc:407] Connection header received
0927 18:42:18.953019 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.953021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.953057 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.953130 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:18.953357 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.953916 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.955068 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.955259 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.955324 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.955603 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.955656 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.955699 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.955791 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.955831 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:42:18.956246 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.952489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40583 (local address 127.18.235.131:51645)
0927 18:42:18.952839 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.952853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.952947 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.953155 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.953158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.953164 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:18.953273 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.953279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.954005 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.954009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.954975 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.954984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.955345 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.955349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.955453 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.955500 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.955722 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.955727 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.955785 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.955861 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":88,"threads_started":1}
I20260927 18:42:18.962368 20787 raft_consensus.cc:1261] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Refusing update from remote peer 0745aba2966147f395559856e2685cae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:18.962604 20647 raft_consensus.cc:1261] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Refusing update from remote peer 0745aba2966147f395559856e2685cae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:18.962777 21027 consensus_queue.cc:1035] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [LEADER]: Connected to new peer: Peer: permanent_uuid: "884b0ece648c471da7201d112a0cdd1e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:18.963052 21004 consensus_queue.cc:1035] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [LEADER]: Connected to new peer: Peer: permanent_uuid: "81288136ac214cfaa3b92b3f1b18ea59" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:18.982702 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.979007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:52837 (local address 127.18.235.131:40097)
0927 18:42:18.979106 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:18.979111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:18.979449 (+ 338us) server_negotiation.cc:407] Connection header received
0927 18:42:18.979544 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:18.979546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:18.979581 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:18.979636 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:18.979841 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.980294 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.981838 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.982040 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.982111 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.982379 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.982440 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.982485 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:18.982577 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 18:42:18.982613 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:42:18.982857 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:18.978871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:40097 (local address 127.18.235.129:52837)
0927 18:42:18.979367 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:18.979380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:18.979477 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:18.979658 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:18.979661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:18.979668 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:18.979759 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.979765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.980378 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.980381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.981741 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:18.981752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:18.982134 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:18.982138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:18.982221 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:18.982268 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:18.982504 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:18.982509 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:42:18.982572 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:42:18.982646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":84,"threads_started":1}
I20260927 18:42:19.005148 20490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:19.005299 20490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:19.007407 20490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e7903b5deec4cee943bd054653bd377: Sending DeleteTablet for 3 replicas of tablet bf9c605134bb42109618eebe4d274cbd
I20260927 18:42:19.008070 20907 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c605134bb42109618eebe4d274cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:19 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:39188
I20260927 18:42:19.008241 21021 tablet_replica.cc:330] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: stopping tablet replica
I20260927 18:42:19.008268 20627 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c605134bb42109618eebe4d274cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:19 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:48274
I20260927 18:42:19.008334 21021 raft_consensus.cc:2229] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:19.008433 21019 tablet_replica.cc:330] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: stopping tablet replica
I20260927 18:42:19.008536 21019 raft_consensus.cc:2229] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:19.008595 21021 raft_consensus.cc:2258] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:19.008627 21019 raft_consensus.cc:2258] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:19.009326 21021 ts_tablet_manager.cc:1905] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:19.009486 21019 ts_tablet_manager.cc:1905] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:19.012069 21021 ts_tablet_manager.cc:1918] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:19.012135 21021 log.cc:1196] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/bf9c605134bb42109618eebe4d274cbd
I20260927 18:42:19.012120 21019 ts_tablet_manager.cc:1918] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:19.012192 21019 log.cc:1196] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/bf9c605134bb42109618eebe4d274cbd
I20260927 18:42:19.012370 21021 ts_tablet_manager.cc:1939] T bf9c605134bb42109618eebe4d274cbd P 0745aba2966147f395559856e2685cae: Deleting consensus metadata
I20260927 18:42:19.012437 21019 ts_tablet_manager.cc:1939] T bf9c605134bb42109618eebe4d274cbd P 81288136ac214cfaa3b92b3f1b18ea59: Deleting consensus metadata
I20260927 18:42:19.013032 20477 catalog_manager.cc:4841] TS 0745aba2966147f395559856e2685cae (127.18.235.131:40097): tablet bf9c605134bb42109618eebe4d274cbd (table test-table [id=ec95fffa5f684311a40b43a58691e813]) successfully deleted
I20260927 18:42:19.013556 20767 tablet_service.cc:1514] Processing DeleteTablet for tablet bf9c605134bb42109618eebe4d274cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:19 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:34180
I20260927 18:42:19.013731 21020 tablet_replica.cc:330] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: stopping tablet replica
I20260927 18:42:19.013835 21020 raft_consensus.cc:2229] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:19.013945 21020 raft_consensus.cc:2258] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:19.014591 20475 catalog_manager.cc:4841] TS 81288136ac214cfaa3b92b3f1b18ea59 (127.18.235.129:40583): tablet bf9c605134bb42109618eebe4d274cbd (table test-table [id=ec95fffa5f684311a40b43a58691e813]) successfully deleted
I20260927 18:42:19.014611 21020 ts_tablet_manager.cc:1905] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:19.017127 21020 ts_tablet_manager.cc:1918] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:19.017189 21020 log.cc:1196] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/bf9c605134bb42109618eebe4d274cbd
I20260927 18:42:19.017397 21020 ts_tablet_manager.cc:1939] T bf9c605134bb42109618eebe4d274cbd P 884b0ece648c471da7201d112a0cdd1e: Deleting consensus metadata
I20260927 18:42:19.018018 20476 catalog_manager.cc:4841] TS 884b0ece648c471da7201d112a0cdd1e (127.18.235.130:38817): tablet bf9c605134bb42109618eebe4d274cbd (table test-table [id=ec95fffa5f684311a40b43a58691e813]) successfully deleted
Sep 27 18:42:19 dist-test-slave-6lr6 krb5kdc[20445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:19.029191 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:19.018013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.18.235.129:40583)
0927 18:42:19.018111 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:19.018116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:19.018128 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:19.018163 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:19.018166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:19.018207 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:19.018288 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:19.019177 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:19.019941 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:19.021177 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:19.021521 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:19.021589 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:19.025403 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:19.025414 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:19.025430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:19.025454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:19.027253 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:19.027701 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:19.027707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:19.027709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:19.027791 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:19.028080 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:19.028083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:19.028085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:19.028355 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:19.028533 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:19.028891 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 18:42:19.029006 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 18:42:19.030218 20687 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48300
I20260927 18:42:19.040040 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:19.031785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.18.235.190:38879)
0927 18:42:19.031909 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:19.031913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:19.032029 (+ 116us) server_negotiation.cc:407] Connection header received
0927 18:42:19.032132 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:19.032135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:19.032184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:19.032261 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:19.032985 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:19.033830 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:19.034906 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:19.035263 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:19.035369 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:19.036795 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:19.036799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:19.036801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:19.036833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:19.038427 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:19.038862 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:19.038865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:19.038867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:19.038947 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:19.039225 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:19.039228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:19.039230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:19.039419 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:19.039539 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:19.039768 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:42:19.039890 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 18:42:19.040560 20490 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53904
I20260927 18:42:19.041985 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20562
I20260927 18:42:19.049847 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20702
I20260927 18:42:19.054744 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20842
I20260927 18:42:19.059610 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 20458
2026-09-27T18:42:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2829 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:19 dist-test-slave-6lr6 krb5kdc[21062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:19 dist-test-slave-6lr6 krb5kdc[21062](info): set up 2 sockets
Sep 27 18:42:19 dist-test-slave-6lr6 krb5kdc[21062](info): commencing operation
krb5kdc: starting...
W20260927 18:42:20.008550 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:20.113433 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:37937
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:37937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:20.215881 21075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:20.216106 21075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:20.216152 21075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:20.220055 21075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:20.220105 21075 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:42:20.220126 21075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:20.220163 21075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:20.220196 21075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:20.220221 21075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:20.225200 21075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:37937
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:37937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:20.226069 21075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:20.226976 21075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:20.234611 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.234615 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.234633 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:20.234827 21075 server_base.cc:1034] running on GCE node
I20260927 18:42:20.235376 21075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:20.236296 21075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:20.237452 21075 hybrid_clock.cc:648] HybridClock initialized: now 1790534540237428 us; error 34 us; skew 500 ppm
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:20.239820 21075 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:20.240609 21075 webserver.cc:460] Webserver started at http://127.18.235.190:46403/ using document root <none> and password file <none>
I20260927 18:42:20.241073 21075 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:20.241170 21075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:20.241360 21075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:20.243157 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "208954f249af4c92afdddc118807e589"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "99838b016ceef2a298292ad9f4efa86e"
server_key_iv: "e723e0afcb29e2dd6299386aa2b246ba"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.243649 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "208954f249af4c92afdddc118807e589"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "99838b016ceef2a298292ad9f4efa86e"
server_key_iv: "e723e0afcb29e2dd6299386aa2b246ba"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.246913 21075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:20.248989 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.250097 21075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:20.250207 21075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "208954f249af4c92afdddc118807e589"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "99838b016ceef2a298292ad9f4efa86e"
server_key_iv: "e723e0afcb29e2dd6299386aa2b246ba"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.250295 21075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:20.264261 21075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:20.266633 21075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:20.266788 21075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:20.273571 21075 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:37937
I20260927 18:42:20.273578 21152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:37937 every 8 connection(s)
I20260927 18:42:20.274369 21075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:20.276772 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:20.277881 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 21075
I20260927 18:42:20.277961 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:20.278187 19374 external_mini_cluster.cc:1428] Setting key b3a9a12b46c4d888b20300f3dec58244
I20260927 18:42:20.282238 21153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Bootstrap starting.
I20260927 18:42:20.284555 21153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:20.285254 21153 log.cc:824] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:20.286826 21153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: No bootstrap required, opened a new log
I20260927 18:42:20.290002 21153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208954f249af4c92afdddc118807e589" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37937 } }
I20260927 18:42:20.290257 21153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:20.290309 21153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208954f249af4c92afdddc118807e589, State: Initialized, Role: FOLLOWER
I20260927 18:42:20.290723 21153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208954f249af4c92afdddc118807e589" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37937 } }
I20260927 18:42:20.290839 21153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:20.290890 21153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:20.290978 21153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:20.291679 21153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208954f249af4c92afdddc118807e589" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37937 } }
I20260927 18:42:20.291970 21153 leader_election.cc:302] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 208954f249af4c92afdddc118807e589; no voters:
I20260927 18:42:20.292204 21153 leader_election.cc:288] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:20.292524 21158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:20.292719 21158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [term 1 LEADER]: Becoming Leader. State: Replica: 208954f249af4c92afdddc118807e589, State: Running, Role: LEADER
I20260927 18:42:20.293004 21158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208954f249af4c92afdddc118807e589" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37937 } }
I20260927 18:42:20.293433 21153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:20.294064 21160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 208954f249af4c92afdddc118807e589. Latest consensus state: current_term: 1 leader_uuid: "208954f249af4c92afdddc118807e589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208954f249af4c92afdddc118807e589" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37937 } } }
I20260927 18:42:20.294169 21160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:20.294737 21159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "208954f249af4c92afdddc118807e589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208954f249af4c92afdddc118807e589" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37937 } } }
I20260927 18:42:20.294845 21159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:20.297003 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.279294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.18.235.190:37937)
0927 18:42:20.279630 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.279645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.279681 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:42:20.280326 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.280343 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.280633 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.280888 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.281702 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.282633 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.283727 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.284158 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.284274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.291512 (+ 7238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.291517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.291528 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.291553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.293392 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.293879 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.293884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.293889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.294015 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.294464 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.294467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.294468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.294935 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:20.295140 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:20.295979 (+ 839us) server_negotiation.cc:299] Negotiation successful
0927 18:42:20.296271 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
W20260927 18:42:20.297837 21174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:20.297895 21174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:20.297951 21166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:20.298702 21166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:20.303650 21166 catalog_manager.cc:1347] Generated new cluster ID: 8439b3b4a9a4472388b679329cf185df
I20260927 18:42:20.303719 21166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:20.324530 21166 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:20.325671 21166 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:20.330567 21166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Generated new TSK 0
I20260927 18:42:20.331248 21166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:20.365341 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37937
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:20.509428 21179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:20.509752 21179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:20.509822 21179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:20.515928 21179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:20.515993 21179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:20.516050 21179 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:42:20.516078 21179 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:20.516124 21179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:20.523901 21179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:20.525147 21179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:20.526427 21179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:20.536873 21184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.536883 21185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.536883 21187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:20.537539 21179 server_base.cc:1034] running on GCE node
I20260927 18:42:20.537966 21179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:20.538690 21179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:20.539870 21179 hybrid_clock.cc:648] HybridClock initialized: now 1790534540539842 us; error 41 us; skew 500 ppm
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:20.543036 21179 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:20.544062 21179 webserver.cc:460] Webserver started at http://127.18.235.129:37545/ using document root <none> and password file <none>
I20260927 18:42:20.544749 21179 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:20.544824 21179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:20.545054 21179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:20.547724 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "035e05e0d57e4377ae63a0b52b91889d"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "e4a5193dcb19e07b50b6c74e436cf8d9"
server_key_iv: "e593004129a5f854807915b2da3eac21"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.548424 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "035e05e0d57e4377ae63a0b52b91889d"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "e4a5193dcb19e07b50b6c74e436cf8d9"
server_key_iv: "e593004129a5f854807915b2da3eac21"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.552958 21179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 18:42:20.555815 21194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.557269 21179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:20.557377 21179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "035e05e0d57e4377ae63a0b52b91889d"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "e4a5193dcb19e07b50b6c74e436cf8d9"
server_key_iv: "e593004129a5f854807915b2da3eac21"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.557483 21179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:20.578358 21179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:20.581653 21179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:20.581830 21179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:20.583544 21179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:20.583601 21179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.583648 21179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:20.583678 21179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:20.594007 21179 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:35689
I20260927 18:42:20.594991 21179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:20.595474 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 21179
I20260927 18:42:20.595584 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:20.595865 19374 external_mini_cluster.cc:1428] Setting key ce8f3317e133ca517a9ced646946d2f3
I20260927 18:42:20.596032 21309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:35689 every 8 connection(s)
I20260927 18:42:20.597174 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.584337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.18.235.129:41835)
0927 18:42:20.584735 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:20.584781 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:20.585628 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:20.586124 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:20.586132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:20.586569 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:20.587165 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.587194 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.588136 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.588140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.589494 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.589505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.590121 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.590125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.590318 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.590856 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:20.590873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:20.592489 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:20.594437 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.594443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.594456 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.594688 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.594939 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.594942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.594944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.595081 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.595630 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:20.595636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:20.595917 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 18:42:20.596101 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 18:42:20.596388 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260927 18:42:20.597934 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.583935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:41835 (local address 127.18.235.190:37937)
0927 18:42:20.584050 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.584054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.584871 (+ 817us) server_negotiation.cc:407] Connection header received
0927 18:42:20.585823 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.585826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.585882 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.585957 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.587323 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.587989 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.589606 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.589967 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.590067 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.592670 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.592674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.592678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.592717 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.594334 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.594777 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.594780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.594782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.594862 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.595158 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.595161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.595163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.595402 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:20.595510 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:20.597741 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0927 18:42:20.597834 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:42:20.598954 21310 heartbeater.cc:347] Connected to a master server at 127.18.235.190:37937
I20260927 18:42:20.599114 21310 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:20.599630 21310 heartbeater.cc:511] Master 127.18.235.190:37937 requested a full tablet report, sending...
I20260927 18:42:20.601349 21106 ts_manager.cc:194] Registered new tserver with Master: 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689)
I20260927 18:42:20.602892 21106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:41835
I20260927 18:42:20.608861 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.600865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:38649 (local address 127.18.235.190:37937)
0927 18:42:20.600949 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.600953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.601226 (+ 273us) server_negotiation.cc:407] Connection header received
0927 18:42:20.601376 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.601378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.601416 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.601459 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.601995 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.602802 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.604093 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.604397 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.604466 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.605707 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.605711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.605714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.605737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.607178 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.607625 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.607628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.607630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.607720 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.608005 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.608009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.608010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.608199 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:20.608267 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:20.608600 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 18:42:20.608715 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":24}
I20260927 18:42:20.609305 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.600922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.18.235.129:38649)
0927 18:42:20.601151 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:20.601166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:20.601288 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:20.601568 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:20.601571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:20.601767 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:20.601875 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.601881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.602926 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.602929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.603970 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.603980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.604595 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.604598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.604681 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.605169 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:20.605182 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:20.605558 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:20.607294 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.607298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.607300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.607519 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.607825 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.607828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.607830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.607911 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.608337 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:20.608340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:20.608423 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 18:42:20.608983 (+ 560us) client_negotiation.cc:240] Negotiation successful
0927 18:42:20.609128 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:20.623579 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37937
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:20.725679 21319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:20.725909 21319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:20.725951 21319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:20.729818 21319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:20.729863 21319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:20.729898 21319 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:42:20.729918 21319 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:20.729946 21319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:20.734817 21319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:20.735688 21319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:20.736548 21319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:20.745164 21324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.745178 21325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.745178 21327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:20.745362 21319 server_base.cc:1034] running on GCE node
I20260927 18:42:20.745786 21319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:20.746363 21319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:20.747527 21319 hybrid_clock.cc:648] HybridClock initialized: now 1790534540747513 us; error 23 us; skew 500 ppm
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:20.750118 21319 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:20.750917 21319 webserver.cc:460] Webserver started at http://127.18.235.130:34261/ using document root <none> and password file <none>
I20260927 18:42:20.751418 21319 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:20.751487 21319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:20.751709 21319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:20.753618 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "4037149a98212820505b7a5de5f25670"
server_key_iv: "1cbcb1b1cb0d8bcdc9e5743c317e9b39"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.754148 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "4037149a98212820505b7a5de5f25670"
server_key_iv: "1cbcb1b1cb0d8bcdc9e5743c317e9b39"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.757571 21319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:42:20.759778 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.760867 21319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:20.760972 21319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "4037149a98212820505b7a5de5f25670"
server_key_iv: "1cbcb1b1cb0d8bcdc9e5743c317e9b39"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.761062 21319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:20.775249 21319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:20.777781 21319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:20.777944 21319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:20.779297 21319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:20.779349 21319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.779403 21319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:20.779431 21319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:20.789438 21319 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42507
I20260927 18:42:20.789500 21449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42507 every 8 connection(s)
I20260927 18:42:20.790217 21319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:20.791663 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.779819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:37835 (local address 127.18.235.190:37937)
0927 18:42:20.779955 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.779960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.781122 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 18:42:20.782173 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.782177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.782231 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.782305 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.783331 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.783844 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.785229 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.785557 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.785638 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.788394 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.788398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.788401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.788426 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.789862 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.790323 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.790326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.790327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.790395 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.790728 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.790731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.790733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.790941 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:20.791035 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:20.791456 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 18:42:20.791546 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:42:20.792341 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.780572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.18.235.130:37835)
0927 18:42:20.780972 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:20.781016 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:20.781975 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:20.782464 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:20.782470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:20.782758 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:20.783208 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.783222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.783953 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.783956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.785059 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.785070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.785827 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.785831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.785996 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.786495 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:20.786509 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:20.788247 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:20.789960 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.789965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.789978 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.790208 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.790473 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.790476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.790478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.790612 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.791042 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:20.791047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:20.791252 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 18:42:20.791415 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:42:20.791641 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260927 18:42:20.793444 21450 heartbeater.cc:347] Connected to a master server at 127.18.235.190:37937
I20260927 18:42:20.793586 21450 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:20.794073 21450 heartbeater.cc:511] Master 127.18.235.190:37937 requested a full tablet report, sending...
I20260927 18:42:20.795177 21107 ts_manager.cc:194] Registered new tserver with Master: 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507)
I20260927 18:42:20.795902 21107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:37835
I20260927 18:42:20.798508 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 21319
I20260927 18:42:20.798605 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:20.798841 19374 external_mini_cluster.cc:1428] Setting key 6a1d3eb0b20b020a7a715077cfd87c5a
I20260927 18:42:20.801870 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.795251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.18.235.130:44873)
0927 18:42:20.795499 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:20.795513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:20.795599 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:20.795801 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:20.795804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:20.796000 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:20.796144 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.796154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.796902 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.796906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.797756 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.797767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.798130 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.798132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.798200 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.798594 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:20.798632 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:20.798979 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:20.800555 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.800557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.800558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.800746 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.800984 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.800986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.800987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.801056 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.801477 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:20.801480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:20.801549 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 18:42:20.801633 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:42:20.801726 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260927 18:42:20.801944 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.795299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:44873 (local address 127.18.235.190:37937)
0927 18:42:20.795403 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.795407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.795580 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:42:20.795663 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.795666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.795703 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.795774 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.796257 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.796778 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.797857 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.798050 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.798117 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.799094 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.799097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.799100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.799122 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.800465 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.800831 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.800834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.800835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.800910 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.801152 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.801154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.801156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.801373 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:20.801467 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:20.801705 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:42:20.801814 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:20.821712 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37937
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:20.923259 21459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:20.923463 21459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:20.923503 21459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:20.927359 21459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:20.927403 21459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:20.927438 21459 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:42:20.927456 21459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:20.927485 21459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:20.932338 21459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:37937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:20.933137 21459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:20.934049 21459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:42:20.942404 21459 server_base.cc:1034] running on GCE node
W20260927 18:42:20.942348 21464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.942324 21467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:20.942330 21465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:20.942948 21459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:20.943481 21459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:20.944628 21459 hybrid_clock.cc:648] HybridClock initialized: now 1790534540944612 us; error 24 us; skew 500 ppm
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:20.946988 21459 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:20.947767 21459 webserver.cc:460] Webserver started at http://127.18.235.131:45063/ using document root <none> and password file <none>
I20260927 18:42:20.948225 21459 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:20.948276 21459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:20.948417 21459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:20.950160 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "d8d3baf50d514d6cbd73206a013478e9"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "e1033a9f214b84e6785d5b2e20ebc620"
server_key_iv: "7b9dcddc13195727335d53d93a7ab80b"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.950616 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "d8d3baf50d514d6cbd73206a013478e9"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "e1033a9f214b84e6785d5b2e20ebc620"
server_key_iv: "7b9dcddc13195727335d53d93a7ab80b"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.953866 21459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:20.955863 21474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.956903 21459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:20.957005 21459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "d8d3baf50d514d6cbd73206a013478e9"
format_stamp: "Formatted at 2026-09-27 18:42:20 on dist-test-slave-6lr6"
server_key: "e1033a9f214b84e6785d5b2e20ebc620"
server_key_iv: "7b9dcddc13195727335d53d93a7ab80b"
server_key_version: "encryptionkey@0"
I20260927 18:42:20.957087 21459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:20.973621 21459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:20.975891 21459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:20.976038 21459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:20.977289 21459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:20.977334 21459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:20.977380 21459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:20.977416 21459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:20 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:20.986598 21459 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:44535
I20260927 18:42:20.986707 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:44535 every 8 connection(s)
I20260927 18:42:20.987432 21459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:20.989424 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.977843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:53731 (local address 127.18.235.190:37937)
0927 18:42:20.977984 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.977990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.979080 (+ 1090us) server_negotiation.cc:407] Connection header received
0927 18:42:20.980126 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.980129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.980198 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.980297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.981518 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.982307 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.983419 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.983641 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.983726 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.986118 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.986121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.986123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.986147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.987492 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.988029 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.988033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.988035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.988125 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.988431 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.988434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.988436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.988608 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:20.988689 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:20.989138 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 18:42:20.989272 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:20.990043 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.978554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.18.235.131:53731)
0927 18:42:20.978939 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:20.978982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:20.979907 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:20.980481 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:20.980486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:20.980871 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:20.981403 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.981415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.982418 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.982422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.983304 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.983323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.983777 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.983781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.983919 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.984359 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:20.984373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:20.985974 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:20.987581 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.987586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.987598 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.987913 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.988214 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.988217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.988219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.988344 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:20.988711 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:20.988717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:20.988910 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 18:42:20.989059 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:42:20.989352 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260927 18:42:20.991003 21590 heartbeater.cc:347] Connected to a master server at 127.18.235.190:37937
I20260927 18:42:20.991124 21590 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:20.991611 21590 heartbeater.cc:511] Master 127.18.235.190:37937 requested a full tablet report, sending...
I20260927 18:42:20.992544 21106 ts_manager.cc:194] Registered new tserver with Master: d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535)
I20260927 18:42:20.993240 21106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:53731
I20260927 18:42:20.996057 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 21459
I20260927 18:42:20.996153 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:20.996521 19374 external_mini_cluster.cc:1428] Setting key cb2910b50b61aecc527771040ac1ec0a
I20260927 18:42:20.998397 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:21.000845 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.992903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.18.235.131:44625)
0927 18:42:20.993170 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:20.993181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:20.993260 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:20.993486 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:20.993489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:20.993670 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:20.993775 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.993781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.994559 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.994561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.995115 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:20.995123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.995533 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.995536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:20.995624 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.995992 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:20.996005 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:20.996320 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:20.997862 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:20.997865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:20.997867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:20.998084 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:21.000029 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:21.000032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:21.000034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:21.000112 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:21.000499 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:21.000501 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:21.000565 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.000633 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.000727 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260927 18:42:21.000946 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:20.992968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:44625 (local address 127.18.235.190:37937)
0927 18:42:20.993061 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:20.993065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:20.993238 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:42:20.993333 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:20.993336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:20.993395 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:20.993465 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:20.993876 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.994421 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.995215 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:20.995444 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:20.995524 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:20.996430 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:20.996434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:20.996436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:20.996458 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:20.997771 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:20.998170 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:20.998172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:20.998174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:20.998262 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:21.000214 (+ 1952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:21.000217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:21.000219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:21.000419 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:21.000513 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.000693 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.000799 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 18:42:21 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:42:21 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:21.021519 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.014853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.18.235.190:37937)
0927 18:42:21.014961 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.014964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.014975 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:21.015004 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.015005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.015038 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.015098 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:21.015680 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.016161 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.016885 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.017079 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.017198 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.019206 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:21.019209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:21.019210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:21.019230 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:21.020414 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:21.020705 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:21.020707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:21.020708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:21.020771 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:21.020938 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:21.020940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:21.020941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:21.021072 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:21.021196 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.021337 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.021415 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:42:21.023749 21106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:21.026602 21106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:21.039197 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.033734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42507 (local address 127.0.0.1:46844)
0927 18:42:21.034173 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.034217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.034327 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.035050 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.035055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.035078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.035202 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.035209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.036717 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.036720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.037938 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.037948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.038712 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.038824 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.038860 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.038958 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.039067 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":61,"thread_start_us":67,"threads_started":1}
I20260927 18:42:21.039207 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.033878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.18.235.130:42507)
0927 18:42:21.034586 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.034597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.034617 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:42:21.034688 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.034693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.034884 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.035351 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.035495 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.036617 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038073 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.038606 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038726 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.038853 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.038994 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.039079 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":91,"threads_started":1}
I20260927 18:42:21.039532 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.034398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:35689 (local address 127.0.0.1:49434)
0927 18:42:21.034592 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.034604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.034679 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.035604 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.035607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.035620 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.035691 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.035696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.036907 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.036911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.037892 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.037902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.039110 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.039114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.039256 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.039278 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.039355 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.039412 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":28,"thread_start_us":75,"threads_started":1}
I20260927 18:42:21.039618 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.034800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.18.235.129:35689)
0927 18:42:21.035185 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.035196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.035217 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:42:21.035292 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.035296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.035478 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.035615 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.035788 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.036816 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038016 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.038807 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038934 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.039050 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.039400 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.039492 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":96,"threads_started":1}
I20260927 18:42:21.039981 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.033874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.0.0.1:35860)
0927 18:42:21.034226 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.034240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.034331 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.035517 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.035520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.035533 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.035616 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.035622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.036553 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.036556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.037938 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.037948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.039110 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.039114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.039231 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.039547 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.039797 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.039868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":308,"thread_start_us":80,"threads_started":1}
I20260927 18:42:21.040158 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.034422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.18.235.131:44535)
0927 18:42:21.034962 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.034974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.034994 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:42:21.035059 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.035064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.035249 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.035412 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.035729 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.036454 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038055 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.038806 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.038938 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.039050 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.039825 (+ 775us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.039907 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":95,"threads_started":1}
I20260927 18:42:21.042141 21384 tablet_service.cc:1467] Processing CreateTablet for tablet 27444d1bea4c4eef9fa828f28233749d (DEFAULT_TABLE table=test-table [id=efb24808d6004f349c439a71180720e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:21.043198 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27444d1bea4c4eef9fa828f28233749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.042956 21524 tablet_service.cc:1467] Processing CreateTablet for tablet 27444d1bea4c4eef9fa828f28233749d (DEFAULT_TABLE table=test-table [id=efb24808d6004f349c439a71180720e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:21.043913 21524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27444d1bea4c4eef9fa828f28233749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.043807 21244 tablet_service.cc:1467] Processing CreateTablet for tablet 27444d1bea4c4eef9fa828f28233749d (DEFAULT_TABLE table=test-table [id=efb24808d6004f349c439a71180720e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:21.044698 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27444d1bea4c4eef9fa828f28233749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.050298 21613 tablet_bootstrap.cc:492] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Bootstrap starting.
I20260927 18:42:21.050856 21615 tablet_bootstrap.cc:492] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Bootstrap starting.
I20260927 18:42:21.052469 21613 tablet_bootstrap.cc:654] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.052800 21615 tablet_bootstrap.cc:654] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.053332 21613 log.cc:824] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:21.053398 21614 tablet_bootstrap.cc:492] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Bootstrap starting.
I20260927 18:42:21.053551 21615 log.cc:824] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:21.055204 21613 tablet_bootstrap.cc:492] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: No bootstrap required, opened a new log
I20260927 18:42:21.055370 21614 tablet_bootstrap.cc:654] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.055387 21613 ts_tablet_manager.cc:1397] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 18:42:21.055390 21615 tablet_bootstrap.cc:492] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: No bootstrap required, opened a new log
I20260927 18:42:21.055552 21615 ts_tablet_manager.cc:1397] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:21.056120 21614 log.cc:824] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:21.057848 21614 tablet_bootstrap.cc:492] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: No bootstrap required, opened a new log
I20260927 18:42:21.058010 21614 ts_tablet_manager.cc:1397] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:21.058302 21613 raft_consensus.cc:348] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.058490 21613 raft_consensus.cc:374] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.058543 21613 raft_consensus.cc:729] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035e05e0d57e4377ae63a0b52b91889d, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.058917 21613 consensus_queue.cc:260] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.058944 21615 raft_consensus.cc:348] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.059172 21615 raft_consensus.cc:374] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.059221 21615 raft_consensus.cc:729] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.059510 21310 heartbeater.cc:503] Master 127.18.235.190:37937 was elected leader, sending a full tablet report...
I20260927 18:42:21.059651 21615 consensus_queue.cc:260] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.059757 21613 ts_tablet_manager.cc:1428] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260927 18:42:21.060307 21590 heartbeater.cc:503] Master 127.18.235.190:37937 was elected leader, sending a full tablet report...
I20260927 18:42:21.060572 21615 ts_tablet_manager.cc:1428] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 18:42:21.061596 21614 raft_consensus.cc:348] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.061751 21614 raft_consensus.cc:374] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.061777 21614 raft_consensus.cc:729] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e1e89a7d5484f89e22fa96b3aaa3f, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.062120 21614 consensus_queue.cc:260] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.062960 21614 ts_tablet_manager.cc:1428] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:21.063230 21450 heartbeater.cc:503] Master 127.18.235.190:37937 was elected leader, sending a full tablet report...
W20260927 18:42:21.097332 21311 tablet.cc:2369] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:21.191236 21620 raft_consensus.cc:482] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:21.191444 21620 raft_consensus.cc:504] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.192584 21620 leader_election.cc:288] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507), 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689)
I20260927 18:42:21.197026 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.192920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:40157 (local address 127.18.235.130:42507)
0927 18:42:21.193046 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.193049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.193062 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:21.193243 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.193247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.193297 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.193373 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.193743 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.194563 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.195895 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.196488 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.196593 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.196635 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.196870 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.196930 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:21.197069 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.192847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42507 (local address 127.18.235.131:40157)
0927 18:42:21.192985 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.193001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.193095 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.193472 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.193475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.193495 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.193605 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.193624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.194679 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.194683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.195772 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.195781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.196594 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.196599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.196738 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.196767 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.196846 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.196919 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 18:42:21.197757 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.193162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:35689 (local address 127.18.235.131:34869)
0927 18:42:21.193539 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.193550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.193631 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.193842 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.193845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.193861 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.194398 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.194406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.195452 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.195456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.196435 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.196444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.197398 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.197401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.197490 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.197511 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.197583 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.197648 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260927 18:42:21.197810 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.193240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:34869 (local address 127.18.235.129:35689)
0927 18:42:21.193382 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.193387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.193623 (+ 236us) server_negotiation.cc:407] Connection header received
0927 18:42:21.193705 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.193709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.193759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.193823 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.194542 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.195328 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.196559 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.197160 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.197277 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.197315 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.197614 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.197693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:42:21.197898 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27444d1bea4c4eef9fa828f28233749d" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" is_pre_election: true
I20260927 18:42:21.198222 21395 raft_consensus.cc:2454] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 0.
I20260927 18:42:21.198390 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27444d1bea4c4eef9fa828f28233749d" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d" is_pre_election: true
I20260927 18:42:21.198675 21264 raft_consensus.cc:2454] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 0.
I20260927 18:42:21.198727 21478 leader_election.cc:302] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275e1e89a7d5484f89e22fa96b3aaa3f, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:21.198997 21620 raft_consensus.cc:2781] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:21.199085 21620 raft_consensus.cc:482] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:21.199134 21620 raft_consensus.cc:3037] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.200183 21620 raft_consensus.cc:504] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.200495 21620 leader_election.cc:288] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 election: Requested vote from peers 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507), 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689)
I20260927 18:42:21.200801 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27444d1bea4c4eef9fa828f28233749d" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
I20260927 18:42:21.200860 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27444d1bea4c4eef9fa828f28233749d" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d"
I20260927 18:42:21.200924 21395 raft_consensus.cc:3037] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.200960 21264 raft_consensus.cc:3037] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.201916 21264 raft_consensus.cc:2454] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 1.
I20260927 18:42:21.201968 21395 raft_consensus.cc:2454] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 1.
I20260927 18:42:21.202183 21478 leader_election.cc:302] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035e05e0d57e4377ae63a0b52b91889d, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:21.202346 21620 raft_consensus.cc:2781] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:21.202546 21620 raft_consensus.cc:686] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 1 LEADER]: Becoming Leader. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Running, Role: LEADER
I20260927 18:42:21.202826 21620 consensus_queue.cc:237] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } }
I20260927 18:42:21.205821 21106 catalog_manager.cc:5495] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131). New cstate: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } health_report { overall_health: UNKNOWN } } }
W20260927 18:42:21.238411 21591 tablet.cc:2369] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:42:21 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534540, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:21.245507 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.238486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37937 (local address 127.0.0.1:58758)
0927 18:42:21.238732 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.238743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.238835 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.239036 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.239039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.239414 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:21.239495 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.239501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.240175 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.240178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.240711 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.240717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.241087 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.241090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.241198 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.241676 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:21.241687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:21.242926 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:21.244322 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:21.244327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:21.244331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:21.244520 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:21.244714 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:21.244716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:21.244717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:21.244778 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:21.245048 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:21.245052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:21.245174 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.245258 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.245364 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260927 18:42:21.245527 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.238550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.18.235.190:37937)
0927 18:42:21.238656 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.238661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.238804 (+ 143us) server_negotiation.cc:407] Connection header received
0927 18:42:21.238910 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.238913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.238949 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.239007 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:21.239595 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.240086 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.240806 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.241003 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.241081 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.243047 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:21.243050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:21.243052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:21.243072 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:21.244248 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:21.244584 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:21.244586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:21.244587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:21.244648 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:21.244836 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:21.244838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:21.244839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:21.244981 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:21.245045 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.245303 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.245416 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":40}
I20260927 18:42:21.248656 21106 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:58758:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:42:21.249332 21106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:21.253525 21524 tablet_service.cc:1467] Processing CreateTablet for tablet cafd2ba651d54aef8cafbbb976f878d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83109a8fbe5043cab3e1067cab13ddd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:21.253541 21384 tablet_service.cc:1467] Processing CreateTablet for tablet cafd2ba651d54aef8cafbbb976f878d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83109a8fbe5043cab3e1067cab13ddd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:21.253384 21244 tablet_service.cc:1467] Processing CreateTablet for tablet cafd2ba651d54aef8cafbbb976f878d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83109a8fbe5043cab3e1067cab13ddd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:21.254065 21524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cafd2ba651d54aef8cafbbb976f878d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.254068 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cafd2ba651d54aef8cafbbb976f878d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.254068 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cafd2ba651d54aef8cafbbb976f878d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.256259 21615 tablet_bootstrap.cc:492] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: Bootstrap starting.
I20260927 18:42:21.256726 21613 tablet_bootstrap.cc:492] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: Bootstrap starting.
I20260927 18:42:21.257185 21615 tablet_bootstrap.cc:654] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.257226 21614 tablet_bootstrap.cc:492] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: Bootstrap starting.
I20260927 18:42:21.257583 21613 tablet_bootstrap.cc:654] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.258014 21614 tablet_bootstrap.cc:654] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.258302 21615 tablet_bootstrap.cc:492] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: No bootstrap required, opened a new log
I20260927 18:42:21.258355 21615 ts_tablet_manager.cc:1397] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:21.258761 21615 raft_consensus.cc:348] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.258852 21615 raft_consensus.cc:374] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.258880 21615 raft_consensus.cc:729] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.258992 21615 consensus_queue.cc:260] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.259049 21614 tablet_bootstrap.cc:492] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: No bootstrap required, opened a new log
I20260927 18:42:21.259114 21614 ts_tablet_manager.cc:1397] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:21.259241 21613 tablet_bootstrap.cc:492] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: No bootstrap required, opened a new log
I20260927 18:42:21.259246 21615 ts_tablet_manager.cc:1428] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:21.259320 21613 ts_tablet_manager.cc:1397] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 18:42:21.259289 21620 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.259404 21620 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:21.259528 21614 raft_consensus.cc:348] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.259614 21614 raft_consensus.cc:374] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.259642 21614 raft_consensus.cc:729] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e1e89a7d5484f89e22fa96b3aaa3f, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.259727 21613 raft_consensus.cc:348] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.259752 21614 consensus_queue.cc:260] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.259821 21613 raft_consensus.cc:374] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.259850 21613 raft_consensus.cc:729] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035e05e0d57e4377ae63a0b52b91889d, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.259953 21613 consensus_queue.cc:260] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.260001 21614 ts_tablet_manager.cc:1428] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:21.260203 21613 ts_tablet_manager.cc:1428] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:21.260001 21621 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.260676 21621 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:21.260783 21619 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.260916 21619 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 18:42:21.291195 21451 tablet.cc:2369] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:21.376924 21620 raft_consensus.cc:482] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:21.377031 21620 raft_consensus.cc:504] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.377373 21620 leader_election.cc:288] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689), 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507)
I20260927 18:42:21.377763 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafd2ba651d54aef8cafbbb976f878d6" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d" is_pre_election: true
I20260927 18:42:21.377779 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafd2ba651d54aef8cafbbb976f878d6" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" is_pre_election: true
I20260927 18:42:21.377918 21395 raft_consensus.cc:2454] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 0.
I20260927 18:42:21.377918 21264 raft_consensus.cc:2454] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 0.
I20260927 18:42:21.378153 21478 leader_election.cc:302] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275e1e89a7d5484f89e22fa96b3aaa3f, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:21.378310 21620 raft_consensus.cc:2781] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:21.378379 21620 raft_consensus.cc:482] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:21.378417 21620 raft_consensus.cc:3037] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.379284 21620 raft_consensus.cc:504] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.379565 21620 leader_election.cc:288] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 election: Requested vote from peers 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689), 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507)
I20260927 18:42:21.379912 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafd2ba651d54aef8cafbbb976f878d6" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
I20260927 18:42:21.379904 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafd2ba651d54aef8cafbbb976f878d6" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d"
I20260927 18:42:21.380021 21395 raft_consensus.cc:3037] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.380021 21264 raft_consensus.cc:3037] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.381021 21395 raft_consensus.cc:2454] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 1.
I20260927 18:42:21.381021 21264 raft_consensus.cc:2454] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 1.
I20260927 18:42:21.381295 21478 leader_election.cc:302] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275e1e89a7d5484f89e22fa96b3aaa3f, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:21.381482 21620 raft_consensus.cc:2781] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:21.381570 21620 raft_consensus.cc:686] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [term 1 LEADER]: Becoming Leader. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Running, Role: LEADER
I20260927 18:42:21.381698 21620 consensus_queue.cc:237] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.382331 21623 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: TxnStatusTablet state changed. Reason: New leader d8d3baf50d514d6cbd73206a013478e9. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.382440 21623 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:21.382711 21635 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:42:21.382933 21635 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:42:21.383220 21106 catalog_manager.cc:5495] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131). New cstate: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } health_report { overall_health: HEALTHY } } }
I20260927 18:42:21.412168 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.408571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.0.0.1:35866)
0927 18:42:21.408714 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.408729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.408823 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.409094 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.409097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.409126 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:21.409223 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.409229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.410018 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.410021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.410774 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.410782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.411195 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.411199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.411292 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.411333 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.411890 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.411903 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.411960 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.412038 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 18:42:21.412168 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.408638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.18.235.131:44535)
0927 18:42:21.408745 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.408749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.408763 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:42:21.408952 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.408954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.409000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.409075 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:21.409326 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.409918 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.410884 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.411105 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.411186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.411455 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.411805 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.411901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.411980 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.412050 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:42:21.417048 21395 raft_consensus.cc:1261] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Refusing update from remote peer d8d3baf50d514d6cbd73206a013478e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:21.417068 21264 raft_consensus.cc:1261] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Refusing update from remote peer d8d3baf50d514d6cbd73206a013478e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:21.417663 21620 consensus_queue.cc:1035] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:21.417824 21623 consensus_queue.cc:1035] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:21.420943 21619 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: TxnStatusTablet state changed. Reason: New leader d8d3baf50d514d6cbd73206a013478e9. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.421048 21619 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:21.421751 21621 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: TxnStatusTablet state changed. Reason: New leader d8d3baf50d514d6cbd73206a013478e9. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.421864 21621 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:21.424194 21623 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.424299 21623 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:21.425372 21620 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.425478 21620 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P d8d3baf50d514d6cbd73206a013478e9: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:21.426546 21619 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.426651 21619 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P 035e05e0d57e4377ae63a0b52b91889d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:21.427524 21640 mvcc.cc:205] Tried to move back new op lower bound from 7334029481638260736 to 7334029481500684288. Current Snapshot: MvccSnapshot[applied={T|T < 7334029481638260736}]
I20260927 18:42:21.428385 21621 tablet_replica.cc:439] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } } }
I20260927 18:42:21.428488 21621 tablet_replica.cc:442] T cafd2ba651d54aef8cafbbb976f878d6 P 275e1e89a7d5484f89e22fa96b3aaa3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:21.436806 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.433775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.18.235.131:44535)
0927 18:42:21.433879 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.433883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.433897 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:42:21.434003 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.434006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.434050 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.434117 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:21.434305 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.435067 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.435843 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.436172 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.436267 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.436437 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.436509 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.436572 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.436629 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.436677 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:42:21.445860 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.443218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:43361 (local address 127.18.235.131:44535)
0927 18:42:21.443324 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.443328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.443339 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:21.443440 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.443443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.443475 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.443527 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:21.443714 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.444161 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.444936 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.445159 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.445240 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.445477 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.445538 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.445590 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.445667 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.445733 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":48}
I20260927 18:42:21.445868 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.443162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.18.235.131:43361)
0927 18:42:21.443272 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.443286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.443360 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.443539 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.443541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.443547 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:21.443636 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.443640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.444247 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.444250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.444842 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.444849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.445239 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.445242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.445323 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.445357 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.445600 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.445605 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.445660 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.445760 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":17}
I20260927 18:42:21.450975 21395 raft_consensus.cc:1261] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Refusing update from remote peer d8d3baf50d514d6cbd73206a013478e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:21.451020 21264 raft_consensus.cc:1261] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Refusing update from remote peer d8d3baf50d514d6cbd73206a013478e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:21.451354 21620 consensus_queue.cc:1035] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:21.451494 21623 consensus_queue.cc:1035] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:21.455097 21639 mvcc.cc:205] Tried to move back new op lower bound from 7334029481780338688 to 7334029480769179648. Current Snapshot: MvccSnapshot[applied={T|T < 7334029481780338688}]
I20260927 18:42:21.484230 21106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:21.484380 21106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:21.486485 21106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Sending DeleteTablet for 3 replicas of tablet 27444d1bea4c4eef9fa828f28233749d
I20260927 18:42:21.487039 21384 tablet_service.cc:1514] Processing DeleteTablet for tablet 27444d1bea4c4eef9fa828f28233749d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:21 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:46844
I20260927 18:42:21.487076 21523 tablet_service.cc:1514] Processing DeleteTablet for tablet 27444d1bea4c4eef9fa828f28233749d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:21 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:35860
I20260927 18:42:21.487242 21244 tablet_service.cc:1514] Processing DeleteTablet for tablet 27444d1bea4c4eef9fa828f28233749d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:21 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49434
I20260927 18:42:21.487470 21654 tablet_replica.cc:330] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: stopping tablet replica
I20260927 18:42:21.487627 21655 tablet_replica.cc:330] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: stopping tablet replica
I20260927 18:42:21.487756 21654 raft_consensus.cc:2229] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:21.487941 21654 raft_consensus.cc:2258] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:21.487944 21655 raft_consensus.cc:2229] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:21.488274 21655 raft_consensus.cc:2258] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:21.488283 21106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:21.488644 21106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:21.488806 21656 tablet_replica.cc:330] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: stopping tablet replica
I20260927 18:42:21.489126 21656 raft_consensus.cc:2229] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:21.489359 21656 raft_consensus.cc:2258] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:21.489756 21655 ts_tablet_manager.cc:1905] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:21.489943 21654 ts_tablet_manager.cc:1905] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:21.490564 21656 ts_tablet_manager.cc:1905] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:21.492569 21654 ts_tablet_manager.cc:1918] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:21.492578 21655 ts_tablet_manager.cc:1918] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:21.492659 21655 log.cc:1196] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/27444d1bea4c4eef9fa828f28233749d
I20260927 18:42:21.492661 21654 log.cc:1196] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/27444d1bea4c4eef9fa828f28233749d
I20260927 18:42:21.492942 21654 ts_tablet_manager.cc:1939] T 27444d1bea4c4eef9fa828f28233749d P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting consensus metadata
I20260927 18:42:21.492971 21655 ts_tablet_manager.cc:1939] T 27444d1bea4c4eef9fa828f28233749d P d8d3baf50d514d6cbd73206a013478e9: Deleting consensus metadata
I20260927 18:42:21.493508 21244 tablet_service.cc:1467] Processing CreateTablet for tablet 75b3bc0a98214e748ebc9455b80fab1d (DEFAULT_TABLE table=test-table [id=7d29e16d99cf432b9931383556bca611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:21.493785 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b3bc0a98214e748ebc9455b80fab1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.493805 21656 ts_tablet_manager.cc:1918] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:21.494007 21656 log.cc:1196] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/27444d1bea4c4eef9fa828f28233749d
I20260927 18:42:21.494098 21094 catalog_manager.cc:4841] TS 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507): tablet 27444d1bea4c4eef9fa828f28233749d (table test-table [id=efb24808d6004f349c439a71180720e3]) successfully deleted
I20260927 18:42:21.494464 21656 ts_tablet_manager.cc:1939] T 27444d1bea4c4eef9fa828f28233749d P 035e05e0d57e4377ae63a0b52b91889d: Deleting consensus metadata
I20260927 18:42:21.495335 21094 catalog_manager.cc:4841] TS 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689): tablet 27444d1bea4c4eef9fa828f28233749d (table test-table [id=efb24808d6004f349c439a71180720e3]) successfully deleted
I20260927 18:42:21.495862 21613 tablet_bootstrap.cc:492] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Bootstrap starting.
I20260927 18:42:21.495914 21093 catalog_manager.cc:4841] TS d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535): tablet 27444d1bea4c4eef9fa828f28233749d (table test-table [id=efb24808d6004f349c439a71180720e3]) successfully deleted
I20260927 18:42:21.496676 21613 tablet_bootstrap.cc:654] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.496621 21384 tablet_service.cc:1467] Processing CreateTablet for tablet 75b3bc0a98214e748ebc9455b80fab1d (DEFAULT_TABLE table=test-table [id=7d29e16d99cf432b9931383556bca611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:21.496843 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b3bc0a98214e748ebc9455b80fab1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.497323 21523 tablet_service.cc:1467] Processing CreateTablet for tablet 75b3bc0a98214e748ebc9455b80fab1d (DEFAULT_TABLE table=test-table [id=7d29e16d99cf432b9931383556bca611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:21.497541 21523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b3bc0a98214e748ebc9455b80fab1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:21.498899 21614 tablet_bootstrap.cc:492] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Bootstrap starting.
I20260927 18:42:21.499275 21615 tablet_bootstrap.cc:492] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Bootstrap starting.
I20260927 18:42:21.499711 21614 tablet_bootstrap.cc:654] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.500079 21615 tablet_bootstrap.cc:654] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:21.500142 21613 tablet_bootstrap.cc:492] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: No bootstrap required, opened a new log
I20260927 18:42:21.500216 21613 ts_tablet_manager.cc:1397] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 18:42:21.500593 21613 raft_consensus.cc:348] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.500700 21613 raft_consensus.cc:374] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.500749 21613 raft_consensus.cc:729] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035e05e0d57e4377ae63a0b52b91889d, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.500860 21613 consensus_queue.cc:260] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.501029 21614 tablet_bootstrap.cc:492] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: No bootstrap required, opened a new log
I20260927 18:42:21.501029 21615 tablet_bootstrap.cc:492] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: No bootstrap required, opened a new log
I20260927 18:42:21.501088 21613 ts_tablet_manager.cc:1428] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:21.501096 21615 ts_tablet_manager.cc:1397] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:21.501096 21614 ts_tablet_manager.cc:1397] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:21.501502 21615 raft_consensus.cc:348] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.501502 21614 raft_consensus.cc:348] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.501597 21615 raft_consensus.cc:374] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.501597 21614 raft_consensus.cc:374] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:21.501628 21615 raft_consensus.cc:729] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.501628 21614 raft_consensus.cc:729] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e1e89a7d5484f89e22fa96b3aaa3f, State: Initialized, Role: FOLLOWER
I20260927 18:42:21.501732 21615 consensus_queue.cc:260] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.501732 21614 consensus_queue.cc:260] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.501967 21615 ts_tablet_manager.cc:1428] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:21.501967 21614 ts_tablet_manager.cc:1428] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:21.552513 21621 raft_consensus.cc:482] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:21.552651 21621 raft_consensus.cc:504] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.553853 21621 leader_election.cc:288] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689), d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535)
I20260927 18:42:21.558213 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.554204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:39669 (local address 127.18.235.131:44535)
0927 18:42:21.554434 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.554438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.554654 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:42:21.554772 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.554775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.554819 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.554882 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.555111 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.555590 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.557090 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.557744 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.557858 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.557907 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.558048 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.558111 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260927 18:42:21.558234 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.554033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.18.235.130:39669)
0927 18:42:21.554575 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.554586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.554678 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.554902 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.554904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.554917 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.555009 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.555015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.555684 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.555687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.556969 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.556979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.557845 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.557847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.557929 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.557949 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.558013 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.558082 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":195,"thread_start_us":95,"threads_started":1}
I20260927 18:42:21.558646 21544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b3bc0a98214e748ebc9455b80fab1d" candidate_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d3baf50d514d6cbd73206a013478e9" is_pre_election: true
I20260927 18:42:21.558890 21544 raft_consensus.cc:2454] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 275e1e89a7d5484f89e22fa96b3aaa3f in term 0.
I20260927 18:42:21.559158 21660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.554033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:35689 (local address 127.18.235.130:49383)
0927 18:42:21.554423 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.554438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.554551 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.554788 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.554791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.554805 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:21.554913 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.554919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.555893 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.555896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.556978 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.556984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.558778 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.558783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.558866 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.558889 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.558966 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.559030 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":78,"threads_started":1}
I20260927 18:42:21.559175 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.554201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:49383 (local address 127.18.235.129:35689)
0927 18:42:21.554340 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.554345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.554520 (+ 175us) server_negotiation.cc:407] Connection header received
0927 18:42:21.554655 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.554658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.554705 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.554770 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:21.555003 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.555784 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.557959 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.558546 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.558650 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.558703 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.558986 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.559057 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:42:21.559286 21337 leader_election.cc:302] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275e1e89a7d5484f89e22fa96b3aaa3f, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:21.559521 21621 raft_consensus.cc:2781] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:21.559571 21621 raft_consensus.cc:482] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:21.559607 21621 raft_consensus.cc:3037] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.559595 21263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b3bc0a98214e748ebc9455b80fab1d" candidate_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d" is_pre_election: true
I20260927 18:42:21.559736 21263 raft_consensus.cc:2454] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 275e1e89a7d5484f89e22fa96b3aaa3f in term 0.
I20260927 18:42:21.560665 21621 raft_consensus.cc:504] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.560969 21621 leader_election.cc:288] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [CANDIDATE]: Term 1 election: Requested vote from peers 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689), d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535)
I20260927 18:42:21.561241 21263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b3bc0a98214e748ebc9455b80fab1d" candidate_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d"
I20260927 18:42:21.561326 21544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b3bc0a98214e748ebc9455b80fab1d" candidate_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d3baf50d514d6cbd73206a013478e9"
I20260927 18:42:21.561358 21263 raft_consensus.cc:3037] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.561434 21544 raft_consensus.cc:3037] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:21.562493 21263 raft_consensus.cc:2454] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 275e1e89a7d5484f89e22fa96b3aaa3f in term 1.
I20260927 18:42:21.562508 21544 raft_consensus.cc:2454] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 275e1e89a7d5484f89e22fa96b3aaa3f in term 1.
I20260927 18:42:21.562830 21337 leader_election.cc:302] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275e1e89a7d5484f89e22fa96b3aaa3f, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:21.563046 21621 raft_consensus.cc:2781] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:21.563232 21621 raft_consensus.cc:686] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 LEADER]: Becoming Leader. State: Replica: 275e1e89a7d5484f89e22fa96b3aaa3f, State: Running, Role: LEADER
I20260927 18:42:21.563467 21621 consensus_queue.cc:237] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:21.565625 21106 catalog_manager.cc:5495] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130). New cstate: current_term: 1 leader_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:21.588830 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.585380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.18.235.130:42507)
0927 18:42:21.585479 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.585483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.585497 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:42:21.585629 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.585632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.585686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.585793 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:21.585962 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.586706 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.587426 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.587762 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.587852 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.588017 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.588526 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.588626 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.588657 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.588712 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:42:21.599195 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.595582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:37441 (local address 127.18.235.131:44535)
0927 18:42:21.595677 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.595681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.595704 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:42:21.595744 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.595747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.595788 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.595860 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:21.596076 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.596810 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.598000 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.598326 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.598420 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.598819 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.598889 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.598955 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.599037 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.599081 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:42:21.599529 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.595327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.18.235.130:37441)
0927 18:42:21.595591 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.595604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.595681 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.595875 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.595877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.595884 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:21.595983 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.595988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.596913 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.596916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.597878 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.597888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.598489 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.598493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.598611 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.598655 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.598968 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.598973 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.599029 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.599099 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260927 18:42:21.605141 21544 raft_consensus.cc:1261] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Refusing update from remote peer 275e1e89a7d5484f89e22fa96b3aaa3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:21.606251 21621 consensus_queue.cc:1035] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:21.606335 21263 raft_consensus.cc:1261] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Refusing update from remote peer 275e1e89a7d5484f89e22fa96b3aaa3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:21.606637 21621 consensus_queue.cc:1035] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:21.620083 21638 mvcc.cc:205] Tried to move back new op lower bound from 7334029482409734144 to 7334029482246467584. Current Snapshot: MvccSnapshot[applied={T|T < 7334029482409734144}]
I20260927 18:42:21.624202 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.619750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:47509 (local address 127.18.235.130:42507)
0927 18:42:21.619839 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:21.619843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:21.619854 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:21.619888 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:21.619890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:21.619929 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:21.619988 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:21.620269 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.621012 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.623291 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.623499 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.623590 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.623844 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.623905 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.623959 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:21.624048 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 18:42:21.624090 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:42:21.624202 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:21.619231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42507 (local address 127.18.235.131:47509)
0927 18:42:21.619432 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:21.619445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:21.619510 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:21.620077 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:21.620080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:21.620087 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:21.620167 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.620172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.622311 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.622314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.623186 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:21.623196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:21.623658 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:21.623662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:21.623723 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:21.623745 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:21.623973 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:21.623978 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:42:21.624030 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 18:42:21.624097 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
W20260927 18:42:21.624853 21639 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:42:23.174854 21105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:23.174985 21105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:23.177055 21105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Sending DeleteTablet for 3 replicas of tablet 75b3bc0a98214e748ebc9455b80fab1d
I20260927 18:42:23.177524 21244 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b3bc0a98214e748ebc9455b80fab1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49434
I20260927 18:42:23.177578 21384 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b3bc0a98214e748ebc9455b80fab1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:46844
I20260927 18:42:23.177660 21523 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b3bc0a98214e748ebc9455b80fab1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:35860
I20260927 18:42:23.177827 21691 tablet_replica.cc:330] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: stopping tablet replica
I20260927 18:42:23.177940 21691 raft_consensus.cc:2229] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:23.178032 21691 raft_consensus.cc:2258] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:23.178213 21692 tablet_replica.cc:330] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: stopping tablet replica
I20260927 18:42:23.178308 21692 raft_consensus.cc:2229] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:23.178565 21692 raft_consensus.cc:2258] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:23.178570 21691 ts_tablet_manager.cc:1905] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:23.179057 21692 ts_tablet_manager.cc:1905] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:23.180598 21691 ts_tablet_manager.cc:1918] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:42:23.180676 21691 log.cc:1196] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/75b3bc0a98214e748ebc9455b80fab1d
I20260927 18:42:23.180900 21691 ts_tablet_manager.cc:1939] T 75b3bc0a98214e748ebc9455b80fab1d P 035e05e0d57e4377ae63a0b52b91889d: Deleting consensus metadata
I20260927 18:42:23.181460 21094 catalog_manager.cc:4841] TS 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689): tablet 75b3bc0a98214e748ebc9455b80fab1d (table test-table [id=7d29e16d99cf432b9931383556bca611]) successfully deleted
I20260927 18:42:23.181530 21692 ts_tablet_manager.cc:1918] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:42:23.181597 21692 log.cc:1196] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/75b3bc0a98214e748ebc9455b80fab1d
I20260927 18:42:23.181835 21692 ts_tablet_manager.cc:1939] T 75b3bc0a98214e748ebc9455b80fab1d P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting consensus metadata
I20260927 18:42:23.182384 21094 catalog_manager.cc:4841] TS 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507): tablet 75b3bc0a98214e748ebc9455b80fab1d (table test-table [id=7d29e16d99cf432b9931383556bca611]) successfully deleted
I20260927 18:42:23.182428 21693 tablet_replica.cc:330] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: stopping tablet replica
I20260927 18:42:23.182525 21693 raft_consensus.cc:2229] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:23.182624 21693 raft_consensus.cc:2258] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:23.183106 21693 ts_tablet_manager.cc:1905] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:23.185693 21693 ts_tablet_manager.cc:1918] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:42:23.185751 21693 log.cc:1196] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/75b3bc0a98214e748ebc9455b80fab1d
I20260927 18:42:23.185950 21693 ts_tablet_manager.cc:1939] T 75b3bc0a98214e748ebc9455b80fab1d P d8d3baf50d514d6cbd73206a013478e9: Deleting consensus metadata
I20260927 18:42:23.186496 21093 catalog_manager.cc:4841] TS d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535): tablet 75b3bc0a98214e748ebc9455b80fab1d (table test-table [id=7d29e16d99cf432b9931383556bca611]) successfully deleted
Sep 27 18:42:23 dist-test-slave-6lr6 krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:42:23 dist-test-slave-6lr6 krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:23.203838 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.196298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.18.235.190:37937)
0927 18:42:23.196546 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.196552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.196564 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:23.196663 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.196665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.196705 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.196774 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:23.197412 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.197943 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.198733 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.199002 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.199073 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.201251 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:23.201255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:23.201257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:23.201278 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:23.202416 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:23.202769 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:23.202773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:23.202775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:23.202856 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:23.203085 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:23.203088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:23.203090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:23.203246 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:23.203357 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.203575 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.203683 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
I20260927 18:42:23.205966 21105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:23.206352 21105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:23.210314 21244 tablet_service.cc:1467] Processing CreateTablet for tablet 63d7246554404f6fbbcd90914fb14538 (DEFAULT_TABLE table=test-table [id=ba560a623dc846a19670eacd949c629b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:23.210348 21384 tablet_service.cc:1467] Processing CreateTablet for tablet 63d7246554404f6fbbcd90914fb14538 (DEFAULT_TABLE table=test-table [id=ba560a623dc846a19670eacd949c629b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:23.210382 21523 tablet_service.cc:1467] Processing CreateTablet for tablet 63d7246554404f6fbbcd90914fb14538 (DEFAULT_TABLE table=test-table [id=ba560a623dc846a19670eacd949c629b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:23.210582 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d7246554404f6fbbcd90914fb14538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:23.210582 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d7246554404f6fbbcd90914fb14538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:23.210644 21523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d7246554404f6fbbcd90914fb14538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:23.212445 21702 tablet_bootstrap.cc:492] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Bootstrap starting.
I20260927 18:42:23.213060 21703 tablet_bootstrap.cc:492] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Bootstrap starting.
I20260927 18:42:23.213356 21702 tablet_bootstrap.cc:654] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:23.213810 21703 tablet_bootstrap.cc:654] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:23.214370 21702 tablet_bootstrap.cc:492] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: No bootstrap required, opened a new log
I20260927 18:42:23.214406 21704 tablet_bootstrap.cc:492] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Bootstrap starting.
I20260927 18:42:23.214457 21702 ts_tablet_manager.cc:1397] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:23.214972 21703 tablet_bootstrap.cc:492] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: No bootstrap required, opened a new log
I20260927 18:42:23.214921 21702 raft_consensus.cc:348] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.215018 21702 raft_consensus.cc:374] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:23.215058 21702 raft_consensus.cc:729] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Initialized, Role: FOLLOWER
I20260927 18:42:23.215173 21704 tablet_bootstrap.cc:654] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:23.215168 21703 ts_tablet_manager.cc:1397] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:23.215198 21702 consensus_queue.cc:260] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.215700 21703 raft_consensus.cc:348] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.215785 21703 raft_consensus.cc:374] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:23.215822 21703 raft_consensus.cc:729] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035e05e0d57e4377ae63a0b52b91889d, State: Initialized, Role: FOLLOWER
I20260927 18:42:23.215868 21702 ts_tablet_manager.cc:1428] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:23.215936 21703 consensus_queue.cc:260] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.216260 21703 ts_tablet_manager.cc:1428] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:23.216557 21704 tablet_bootstrap.cc:492] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: No bootstrap required, opened a new log
I20260927 18:42:23.216629 21704 ts_tablet_manager.cc:1397] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:23.216935 21704 raft_consensus.cc:348] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.217020 21704 raft_consensus.cc:374] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:23.217047 21704 raft_consensus.cc:729] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e1e89a7d5484f89e22fa96b3aaa3f, State: Initialized, Role: FOLLOWER
I20260927 18:42:23.217178 21704 consensus_queue.cc:260] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.217449 21704 ts_tablet_manager.cc:1428] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:23.501803 21708 raft_consensus.cc:482] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:23.501928 21708 raft_consensus.cc:504] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.502925 21708 leader_election.cc:288] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507), d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535)
I20260927 18:42:23.506613 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.503090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42507 (local address 127.18.235.129:35117)
0927 18:42:23.503431 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:23.503446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:23.503558 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:23.503789 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:23.503792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:23.503808 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:23.503970 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.503976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.504694 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.504697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.505545 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.505554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.506167 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.506171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.506286 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.506313 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:42:23.506373 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 18:42:23.506438 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":42,"thread_start_us":112,"threads_started":1}
I20260927 18:42:23.506603 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.503189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:35117 (local address 127.18.235.130:42507)
0927 18:42:23.503444 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.503454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.503526 (+ 72us) server_negotiation.cc:407] Connection header received
0927 18:42:23.503658 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.503660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.503701 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.503769 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:23.504071 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.504594 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.505661 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.506065 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.506167 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.506219 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.506401 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.506467 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260927 18:42:23.506611 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.503227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:59203 (local address 127.18.235.131:44535)
0927 18:42:23.503498 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.503508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.503581 (+ 73us) server_negotiation.cc:407] Connection header received
0927 18:42:23.503709 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.503713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.503753 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.503823 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:23.504116 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.504621 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.505657 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.506053 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.506157 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.506224 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.506420 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.506489 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260927 18:42:23.506628 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.503144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.18.235.129:59203)
0927 18:42:23.503495 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:23.503509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:23.503614 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:23.503844 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:23.503847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:23.503862 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:23.504014 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.504019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.504752 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.504755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.505550 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.505558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.506158 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.506162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.506296 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.506331 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:42:23.506391 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 18:42:23.506455 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":65,"thread_start_us":128,"threads_started":1}
I20260927 18:42:23.506976 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d7246554404f6fbbcd90914fb14538" candidate_uuid: "035e05e0d57e4377ae63a0b52b91889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" is_pre_election: true
I20260927 18:42:23.507107 21395 raft_consensus.cc:2454] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035e05e0d57e4377ae63a0b52b91889d in term 0.
I20260927 18:42:23.507164 21544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d7246554404f6fbbcd90914fb14538" candidate_uuid: "035e05e0d57e4377ae63a0b52b91889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d3baf50d514d6cbd73206a013478e9" is_pre_election: true
I20260927 18:42:23.507287 21544 raft_consensus.cc:2454] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035e05e0d57e4377ae63a0b52b91889d in term 0.
I20260927 18:42:23.507402 21198 leader_election.cc:302] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035e05e0d57e4377ae63a0b52b91889d, 275e1e89a7d5484f89e22fa96b3aaa3f; no voters:
I20260927 18:42:23.507644 21708 raft_consensus.cc:2781] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:23.507706 21708 raft_consensus.cc:482] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:23.507748 21708 raft_consensus.cc:3037] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:23.508670 21708 raft_consensus.cc:504] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.508970 21708 leader_election.cc:288] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [CANDIDATE]: Term 1 election: Requested vote from peers 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507), d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535)
I20260927 18:42:23.509279 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d7246554404f6fbbcd90914fb14538" candidate_uuid: "035e05e0d57e4377ae63a0b52b91889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
I20260927 18:42:23.509323 21544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d7246554404f6fbbcd90914fb14538" candidate_uuid: "035e05e0d57e4377ae63a0b52b91889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d3baf50d514d6cbd73206a013478e9"
I20260927 18:42:23.509390 21395 raft_consensus.cc:3037] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:23.509406 21544 raft_consensus.cc:3037] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:23.510411 21544 raft_consensus.cc:2454] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035e05e0d57e4377ae63a0b52b91889d in term 1.
I20260927 18:42:23.510411 21395 raft_consensus.cc:2454] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035e05e0d57e4377ae63a0b52b91889d in term 1.
I20260927 18:42:23.510651 21197 leader_election.cc:302] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035e05e0d57e4377ae63a0b52b91889d, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:23.510807 21708 raft_consensus.cc:2781] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:23.510995 21708 raft_consensus.cc:686] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 1 LEADER]: Becoming Leader. State: Replica: 035e05e0d57e4377ae63a0b52b91889d, State: Running, Role: LEADER
I20260927 18:42:23.511206 21708 consensus_queue.cc:237] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.513167 21106 catalog_manager.cc:5495] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d reported cstate change: term changed from 0 to 1, leader changed from <none> to 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129). New cstate: current_term: 1 leader_uuid: "035e05e0d57e4377ae63a0b52b91889d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:23.578898 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.575791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.18.235.129:35689)
0927 18:42:23.576021 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.576027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.576038 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:23.576100 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.576102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.576148 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.576243 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:23.576411 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.576939 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.577717 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.577922 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.577985 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.578155 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:23.578574 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:23.578654 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.578709 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.578750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260927 18:42:23.587160 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.584687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:41827 (local address 127.18.235.131:44535)
0927 18:42:23.584798 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.584802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.584859 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:42:23.584946 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.584948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.584985 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.585041 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:23.585231 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.585700 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.586417 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.586609 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.586670 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.586876 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:23.586930 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:23.586975 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.587031 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.587070 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:42:23.587189 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.584524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44535 (local address 127.18.235.129:41827)
0927 18:42:23.584794 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:23.584806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:23.584883 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:23.585050 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:23.585052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:23.585057 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:23.585158 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.585162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.585772 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.585774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.586340 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.586346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.586675 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.586678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.586751 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.586784 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:23.586982 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:23.586986 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:42:23.587025 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 18:42:23.587080 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260927 18:42:23.591997 21544 raft_consensus.cc:1261] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Refusing update from remote peer 035e05e0d57e4377ae63a0b52b91889d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:23.591997 21395 raft_consensus.cc:1261] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Refusing update from remote peer 035e05e0d57e4377ae63a0b52b91889d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:23.592397 21713 consensus_queue.cc:1035] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [LEADER]: Connected to new peer: Peer: permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:23.592552 21708 consensus_queue.cc:1035] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:23.613876 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.610426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:45927 (local address 127.18.235.129:35689)
0927 18:42:23.610524 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.610528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.610674 (+ 146us) server_negotiation.cc:407] Connection header received
0927 18:42:23.610822 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.610825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.610871 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.610934 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:23.611257 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.611746 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.612894 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.613185 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.613262 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.613501 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:23.613579 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:23.613643 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.613743 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.613787 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:42:23.613905 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.610379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:35689 (local address 127.18.235.131:45927)
0927 18:42:23.610597 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:23.610609 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:23.610702 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:23.611029 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:23.611032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:23.611039 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:23.611162 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.611168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.611827 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.611830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.612785 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:23.612795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.613278 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.613282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:23.613376 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.613408 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:23.613662 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:23.613668 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:42:23.613725 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:42:23.613801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260927 18:42:23.630517 21105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:23.630646 21105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:23.632373 21105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Sending DeleteTablet for 3 replicas of tablet 63d7246554404f6fbbcd90914fb14538
I20260927 18:42:23.632900 21244 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d7246554404f6fbbcd90914fb14538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49434
I20260927 18:42:23.632977 21384 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d7246554404f6fbbcd90914fb14538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:46844
I20260927 18:42:23.633046 21691 tablet_replica.cc:330] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: stopping tablet replica
I20260927 18:42:23.633150 21692 tablet_replica.cc:330] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: stopping tablet replica
I20260927 18:42:23.633173 21691 raft_consensus.cc:2229] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:23.633249 21692 raft_consensus.cc:2229] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:23.633342 21692 raft_consensus.cc:2258] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:23.633427 21691 raft_consensus.cc:2258] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:23.633431 21105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:23.633792 21105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:23.633785 21523 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d7246554404f6fbbcd90914fb14538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:35860
I20260927 18:42:23.633940 21693 tablet_replica.cc:330] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: stopping tablet replica
I20260927 18:42:23.634057 21693 raft_consensus.cc:2229] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:23.634151 21693 raft_consensus.cc:2258] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:23.634347 21692 ts_tablet_manager.cc:1905] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:23.634770 21693 ts_tablet_manager.cc:1905] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:23.635772 21691 ts_tablet_manager.cc:1905] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:23.636790 21692 ts_tablet_manager.cc:1918] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:23.636852 21692 log.cc:1196] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/63d7246554404f6fbbcd90914fb14538
I20260927 18:42:23.637065 21692 ts_tablet_manager.cc:1939] T 63d7246554404f6fbbcd90914fb14538 P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting consensus metadata
I20260927 18:42:23.637178 21693 ts_tablet_manager.cc:1918] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:23.637243 21693 log.cc:1196] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/63d7246554404f6fbbcd90914fb14538
I20260927 18:42:23.637467 21693 ts_tablet_manager.cc:1939] T 63d7246554404f6fbbcd90914fb14538 P d8d3baf50d514d6cbd73206a013478e9: Deleting consensus metadata
I20260927 18:42:23.637645 21094 catalog_manager.cc:4841] TS 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507): tablet 63d7246554404f6fbbcd90914fb14538 (table test-table [id=ba560a623dc846a19670eacd949c629b]) successfully deleted
I20260927 18:42:23.637989 21093 catalog_manager.cc:4841] TS d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535): tablet 63d7246554404f6fbbcd90914fb14538 (table test-table [id=ba560a623dc846a19670eacd949c629b]) successfully deleted
I20260927 18:42:23.638036 21691 ts_tablet_manager.cc:1918] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:23.638088 21691 log.cc:1196] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/63d7246554404f6fbbcd90914fb14538
I20260927 18:42:23.638304 21691 ts_tablet_manager.cc:1939] T 63d7246554404f6fbbcd90914fb14538 P 035e05e0d57e4377ae63a0b52b91889d: Deleting consensus metadata
I20260927 18:42:23.638779 21094 catalog_manager.cc:4841] TS 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689): tablet 63d7246554404f6fbbcd90914fb14538 (table test-table [id=ba560a623dc846a19670eacd949c629b]) successfully deleted
I20260927 18:42:23.640961 21244 tablet_service.cc:1467] Processing CreateTablet for tablet dcb07ca98f6b417abf5c203bc5c267e2 (DEFAULT_TABLE table=test-table [id=f0fd1eed6e2848dda8ff77e3dc21f3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:23.641139 21384 tablet_service.cc:1467] Processing CreateTablet for tablet dcb07ca98f6b417abf5c203bc5c267e2 (DEFAULT_TABLE table=test-table [id=f0fd1eed6e2848dda8ff77e3dc21f3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:23.641196 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07ca98f6b417abf5c203bc5c267e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:23.641211 21523 tablet_service.cc:1467] Processing CreateTablet for tablet dcb07ca98f6b417abf5c203bc5c267e2 (DEFAULT_TABLE table=test-table [id=f0fd1eed6e2848dda8ff77e3dc21f3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:23.641345 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07ca98f6b417abf5c203bc5c267e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:23.641412 21523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb07ca98f6b417abf5c203bc5c267e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:23.643157 21703 tablet_bootstrap.cc:492] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d: Bootstrap starting.
I20260927 18:42:23.643256 21704 tablet_bootstrap.cc:492] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Bootstrap starting.
I20260927 18:42:23.643337 21702 tablet_bootstrap.cc:492] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Bootstrap starting.
I20260927 18:42:23.643956 21704 tablet_bootstrap.cc:654] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:23.643981 21703 tablet_bootstrap.cc:654] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:23.643981 21702 tablet_bootstrap.cc:654] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:23.645082 21702 tablet_bootstrap.cc:492] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: No bootstrap required, opened a new log
I20260927 18:42:23.645176 21702 ts_tablet_manager.cc:1397] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:23.645316 21703 tablet_bootstrap.cc:492] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d: No bootstrap required, opened a new log
I20260927 18:42:23.645381 21703 ts_tablet_manager.cc:1397] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:23.645455 21702 raft_consensus.cc:348] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.645526 21702 raft_consensus.cc:374] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:23.645545 21702 raft_consensus.cc:729] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Initialized, Role: FOLLOWER
I20260927 18:42:23.645632 21702 consensus_queue.cc:260] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.645738 21703 raft_consensus.cc:348] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.645836 21703 raft_consensus.cc:374] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:23.645864 21703 raft_consensus.cc:729] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035e05e0d57e4377ae63a0b52b91889d, State: Initialized, Role: FOLLOWER
I20260927 18:42:23.645860 21702 ts_tablet_manager.cc:1428] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:42:23.645958 21703 consensus_queue.cc:260] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.646180 21703 ts_tablet_manager.cc:1428] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:23.646776 21704 tablet_bootstrap.cc:492] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: No bootstrap required, opened a new log
I20260927 18:42:23.646858 21704 ts_tablet_manager.cc:1397] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 18:42:23.647202 21704 raft_consensus.cc:348] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.647302 21704 raft_consensus.cc:374] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:23.647332 21704 raft_consensus.cc:729] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275e1e89a7d5484f89e22fa96b3aaa3f, State: Initialized, Role: FOLLOWER
I20260927 18:42:23.647442 21704 consensus_queue.cc:260] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.647679 21704 ts_tablet_manager.cc:1428] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:23.832332 21672 raft_consensus.cc:482] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:23.832454 21672 raft_consensus.cc:504] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.832820 21672 leader_election.cc:288] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689), 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507)
I20260927 18:42:23.833309 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07ca98f6b417abf5c203bc5c267e2" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" is_pre_election: true
I20260927 18:42:23.833309 21263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07ca98f6b417abf5c203bc5c267e2" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d" is_pre_election: true
I20260927 18:42:23.833459 21263 raft_consensus.cc:2454] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 0.
I20260927 18:42:23.833459 21395 raft_consensus.cc:2454] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 0.
I20260927 18:42:23.833727 21478 leader_election.cc:302] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035e05e0d57e4377ae63a0b52b91889d, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:23.833881 21672 raft_consensus.cc:2781] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:23.833940 21672 raft_consensus.cc:482] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:23.833984 21672 raft_consensus.cc:3037] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:23.834909 21672 raft_consensus.cc:504] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.835187 21672 leader_election.cc:288] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 election: Requested vote from peers 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689), 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507)
I20260927 18:42:23.835497 21263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07ca98f6b417abf5c203bc5c267e2" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035e05e0d57e4377ae63a0b52b91889d"
I20260927 18:42:23.835554 21395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb07ca98f6b417abf5c203bc5c267e2" candidate_uuid: "d8d3baf50d514d6cbd73206a013478e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f"
I20260927 18:42:23.835611 21263 raft_consensus.cc:3037] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:23.835672 21395 raft_consensus.cc:3037] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:23.836661 21263 raft_consensus.cc:2454] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 1.
I20260927 18:42:23.836668 21395 raft_consensus.cc:2454] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d3baf50d514d6cbd73206a013478e9 in term 1.
I20260927 18:42:23.836920 21478 leader_election.cc:302] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035e05e0d57e4377ae63a0b52b91889d, d8d3baf50d514d6cbd73206a013478e9; no voters:
I20260927 18:42:23.837086 21672 raft_consensus.cc:2781] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:23.837213 21672 raft_consensus.cc:686] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 1 LEADER]: Becoming Leader. State: Replica: d8d3baf50d514d6cbd73206a013478e9, State: Running, Role: LEADER
I20260927 18:42:23.837344 21672 consensus_queue.cc:237] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } }
I20260927 18:42:23.838819 21105 catalog_manager.cc:5495] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131). New cstate: current_term: 1 leader_uuid: "d8d3baf50d514d6cbd73206a013478e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d3baf50d514d6cbd73206a013478e9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44535 } health_report { overall_health: HEALTHY } } }
I20260927 18:42:23.885444 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:23.883009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.18.235.131:44535)
0927 18:42:23.883108 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:23.883112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:23.883124 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:23.883239 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:23.883241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:23.883279 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:23.883335 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:23.883487 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.883965 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.884676 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:23.884873 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:23.884937 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:23.885093 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:23.885184 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:23.885243 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:23.885309 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 18:42:23.885350 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:42:23.890379 21263 raft_consensus.cc:1261] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d [term 1 FOLLOWER]: Refusing update from remote peer d8d3baf50d514d6cbd73206a013478e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:23.890455 21395 raft_consensus.cc:1261] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Refusing update from remote peer d8d3baf50d514d6cbd73206a013478e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:23.890748 21653 consensus_queue.cc:1035] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275e1e89a7d5484f89e22fa96b3aaa3f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:23.890895 21677 consensus_queue.cc:1035] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035e05e0d57e4377ae63a0b52b91889d" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:42:23.902416 21670 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:42:25.445533 21105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:25.445667 21105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:25.447738 21105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 208954f249af4c92afdddc118807e589: Sending DeleteTablet for 3 replicas of tablet dcb07ca98f6b417abf5c203bc5c267e2
I20260927 18:42:25.448200 21244 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb07ca98f6b417abf5c203bc5c267e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:25 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49434
I20260927 18:42:25.448259 21384 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb07ca98f6b417abf5c203bc5c267e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:25 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:46844
I20260927 18:42:25.448509 21752 tablet_replica.cc:330] T dcb07ca98f6b417abf5c203bc5c267e2 P 035e05e0d57e4377ae63a0b52b91889d: stopping tablet replica
I20260927 18:42:25.448523 21523 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb07ca98f6b417abf5c203bc5c267e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:25 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:35860
I20260927 18:42:25.448601 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 21179
I20260927 18:42:25.449261 21754 tablet_replica.cc:330] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: stopping tablet replica
I20260927 18:42:25.449368 21754 raft_consensus.cc:2229] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:25.449491 21754 raft_consensus.cc:2258] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:25.449951 21754 ts_tablet_manager.cc:1905] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:25.448659 21753 tablet_replica.cc:330] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: stopping tablet replica
I20260927 18:42:25.450250 21753 raft_consensus.cc:2229] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:25.450358 21753 raft_consensus.cc:2258] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:25.450901 21753 ts_tablet_manager.cc:1905] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:25.451941 21754 ts_tablet_manager.cc:1918] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:42:25.451988 21754 log.cc:1196] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/dcb07ca98f6b417abf5c203bc5c267e2
I20260927 18:42:25.452157 21754 ts_tablet_manager.cc:1939] T dcb07ca98f6b417abf5c203bc5c267e2 P d8d3baf50d514d6cbd73206a013478e9: Deleting consensus metadata
I20260927 18:42:25.452984 21093 catalog_manager.cc:4841] TS d8d3baf50d514d6cbd73206a013478e9 (127.18.235.131:44535): tablet dcb07ca98f6b417abf5c203bc5c267e2 (table test-table [id=f0fd1eed6e2848dda8ff77e3dc21f3bc]) successfully deleted
I20260927 18:42:25.453305 21753 ts_tablet_manager.cc:1918] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:42:25.453374 21753 log.cc:1196] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/dcb07ca98f6b417abf5c203bc5c267e2
I20260927 18:42:25.453593 21753 ts_tablet_manager.cc:1939] T dcb07ca98f6b417abf5c203bc5c267e2 P 275e1e89a7d5484f89e22fa96b3aaa3f: Deleting consensus metadata
I20260927 18:42:25.454185 21094 catalog_manager.cc:4841] TS 275e1e89a7d5484f89e22fa96b3aaa3f (127.18.235.130:42507): tablet dcb07ca98f6b417abf5c203bc5c267e2 (table test-table [id=f0fd1eed6e2848dda8ff77e3dc21f3bc]) successfully deleted
I20260927 18:42:25.456163 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 21319
W20260927 18:42:25.456166 21094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:35689 (error 108)
I20260927 18:42:25.457523 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:25.457140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:35689 (local address 127.0.0.1:38234)
0927 18:42:25.457374 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:25.457450 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
W20260927 18:42:25.457638 21094 catalog_manager.cc:4584] TS 035e05e0d57e4377ae63a0b52b91889d (127.18.235.129:35689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcb07ca98f6b417abf5c203bc5c267e2: Network error: Client connection negotiation failed: client connection to 127.18.235.129:35689: connect: Connection refused (error 111)
I20260927 18:42:25.462312 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 21459
I20260927 18:42:25.467629 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 21075
2026-09-27T18:42:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6407 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:25 dist-test-slave-6lr6 krb5kdc[21759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:25 dist-test-slave-6lr6 krb5kdc[21759](info): set up 2 sockets
Sep 27 18:42:25 dist-test-slave-6lr6 krb5kdc[21759](info): commencing operation
krb5kdc: starting...
W20260927 18:42:26.416635 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:26 dist-test-slave-6lr6 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:26.525705 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:33523
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:26.629796 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:26.630007 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:26.630048 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:26.633872 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:26.633914 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:26.633934 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:26.633963 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:26.633988 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:26.638908 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:33523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:26.639767 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:26.640654 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:26.648551 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:26.648543 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:26.648526 21772 server_base.cc:1034] running on GCE node
W20260927 18:42:26.648589 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:26.649243 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:26.650206 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:26.651355 21772 hybrid_clock.cc:648] HybridClock initialized: now 1790534546651341 us; error 25 us; skew 500 ppm
Sep 27 18:42:26 dist-test-slave-6lr6 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534546, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:26.653765 21772 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:26.654523 21772 webserver.cc:460] Webserver started at http://127.18.235.190:36181/ using document root <none> and password file <none>
I20260927 18:42:26.655000 21772 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:26.655054 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:26.655237 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:26.657076 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b"
format_stamp: "Formatted at 2026-09-27 18:42:26 on dist-test-slave-6lr6"
server_key: "43632d37b4da4653cc9b17dac650d207"
server_key_iv: "854b44610174473d012acd92c8ed733b"
server_key_version: "encryptionkey@0"
I20260927 18:42:26.657609 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b"
format_stamp: "Formatted at 2026-09-27 18:42:26 on dist-test-slave-6lr6"
server_key: "43632d37b4da4653cc9b17dac650d207"
server_key_iv: "854b44610174473d012acd92c8ed733b"
server_key_version: "encryptionkey@0"
I20260927 18:42:26.660887 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:42:26.663095 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:26.664451 21772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:26.664569 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b"
format_stamp: "Formatted at 2026-09-27 18:42:26 on dist-test-slave-6lr6"
server_key: "43632d37b4da4653cc9b17dac650d207"
server_key_iv: "854b44610174473d012acd92c8ed733b"
server_key_version: "encryptionkey@0"
I20260927 18:42:26.664677 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:26.681006 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:26.683573 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:26.683755 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:26.690652 21772 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:33523
I20260927 18:42:26.690656 21839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:33523 every 8 connection(s)
I20260927 18:42:26.691540 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:26.694280 21840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:26.699971 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 21772
I20260927 18:42:26.700075 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:26.700343 19374 external_mini_cluster.cc:1428] Setting key 6949071d9ef06c79e6b13df0ec7af82d
I20260927 18:42:26.700892 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b: Bootstrap starting.
I20260927 18:42:26.703454 21840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:26.704144 21840 log.cc:824] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:26.706071 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b: No bootstrap required, opened a new log
I20260927 18:42:26.710026 21840 raft_consensus.cc:348] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33523 } }
I20260927 18:42:26.710281 21840 raft_consensus.cc:374] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:26.710362 21840 raft_consensus.cc:729] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d549c8a14b4c46e5bc7bc3c76bddf78b, State: Initialized, Role: FOLLOWER
I20260927 18:42:26.710778 21840 consensus_queue.cc:260] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33523 } }
Sep 27 18:42:26 dist-test-slave-6lr6 krb5kdc[21759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:26.710896 21840 raft_consensus.cc:388] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:26.710958 21840 raft_consensus.cc:482] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:26.711045 21840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:26.711822 21840 raft_consensus.cc:504] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33523 } }
I20260927 18:42:26.712126 21840 leader_election.cc:302] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d549c8a14b4c46e5bc7bc3c76bddf78b; no voters:
I20260927 18:42:26.712406 21840 leader_election.cc:288] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:26.712495 21845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:26.712731 21845 raft_consensus.cc:686] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [term 1 LEADER]: Becoming Leader. State: Replica: d549c8a14b4c46e5bc7bc3c76bddf78b, State: Running, Role: LEADER
I20260927 18:42:26.713084 21845 consensus_queue.cc:237] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33523 } }
I20260927 18:42:26.713542 21840 sys_catalog.cc:564] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:26.714725 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d549c8a14b4c46e5bc7bc3c76bddf78b. Latest consensus state: current_term: 1 leader_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33523 } } }
I20260927 18:42:26.714836 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:26.715130 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d549c8a14b4c46e5bc7bc3c76bddf78b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33523 } } }
I20260927 18:42:26.715224 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:26.717178 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:26.702543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.18.235.190:33523)
0927 18:42:26.702975 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:26.702998 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:26.703044 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:42:26.703882 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:26.703902 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:26.704219 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:26.704544 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:26.705382 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:26.706523 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:26.707673 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:26.708412 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:26.708579 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:26.711415 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:26.711420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:26.711432 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:26.711456 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:26.713393 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:26.713864 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:26.713870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:26.713875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:26.714005 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:26.714316 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:26.714319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:26.714321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:26.714810 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:26.715007 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:26.715971 (+ 964us) server_negotiation.cc:299] Negotiation successful
0927 18:42:26.716264 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
W20260927 18:42:26.717942 21860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:26.718015 21860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:26.718068 21857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:26.718861 21857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:26.724017 21857 catalog_manager.cc:1347] Generated new cluster ID: 474b40d36a5a47a48aa17e7811813457
I20260927 18:42:26.724073 21857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:26.761626 21857 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:26.762912 21857 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:26.771008 21857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d549c8a14b4c46e5bc7bc3c76bddf78b: Generated new TSK 0
I20260927 18:42:26.771632 21857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:26.804903 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33523
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:26.909288 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:26.909514 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:26.909567 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:26.913409 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:26.913452 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:26.913527 21866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:26.918486 21866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:26.919337 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:26.920220 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:26.929076 21874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:26.929064 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:26.929056 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:26.929124 21866 server_base.cc:1034] running on GCE node
I20260927 18:42:26.929679 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:26.930218 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:26.931365 21866 hybrid_clock.cc:648] HybridClock initialized: now 1790534546931349 us; error 27 us; skew 500 ppm
Sep 27 18:42:26 dist-test-slave-6lr6 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534546, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:26.933871 21866 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:26.934633 21866 webserver.cc:460] Webserver started at http://127.18.235.129:35177/ using document root <none> and password file <none>
I20260927 18:42:26.935166 21866 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:26.935233 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:26.935429 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:26.937282 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "04185c99c31244e0912adf534847a08a"
format_stamp: "Formatted at 2026-09-27 18:42:26 on dist-test-slave-6lr6"
server_key: "2975d5ea649b5783cfd6361050c2707b"
server_key_iv: "978ff3851371c2c06350bf5843770ee7"
server_key_version: "encryptionkey@0"
I20260927 18:42:26.937774 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "04185c99c31244e0912adf534847a08a"
format_stamp: "Formatted at 2026-09-27 18:42:26 on dist-test-slave-6lr6"
server_key: "2975d5ea649b5783cfd6361050c2707b"
server_key_iv: "978ff3851371c2c06350bf5843770ee7"
server_key_version: "encryptionkey@0"
I20260927 18:42:26.940945 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:26.943009 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:26.944016 21866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:26.944110 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "04185c99c31244e0912adf534847a08a"
format_stamp: "Formatted at 2026-09-27 18:42:26 on dist-test-slave-6lr6"
server_key: "2975d5ea649b5783cfd6361050c2707b"
server_key_iv: "978ff3851371c2c06350bf5843770ee7"
server_key_version: "encryptionkey@0"
I20260927 18:42:26.944190 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:26.970647 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:26.972922 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:26.973066 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:26.973547 21866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:26.974233 21866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:26.974277 21866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:26.974321 21866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:26.974350 21866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:26.981953 21866 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:36297
I20260927 18:42:26.981982 21994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:36297 every 8 connection(s)
I20260927 18:42:26.982657 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:26.990056 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 21866
I20260927 18:42:26.990163 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:26.990414 19374 external_mini_cluster.cc:1428] Setting key 035fffc04eb17da9e5fc1c3a7ae85a51
Sep 27 18:42:26 dist-test-slave-6lr6 krb5kdc[21759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534546, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:26.994270 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:26.984132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:45361 (local address 127.18.235.190:33523)
0927 18:42:26.984240 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:26.984244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:26.985162 (+ 918us) server_negotiation.cc:407] Connection header received
0927 18:42:26.985947 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:26.985950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:26.986000 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:26.986082 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:26.987069 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:26.987590 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:26.988619 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:26.988835 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:26.988902 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:26.991176 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:26.991179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:26.991181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:26.991214 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:26.992434 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:26.992961 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:26.992964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:26.992966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:26.993052 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:26.993347 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:26.993349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:26.993351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:26.993542 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:26.993630 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:26.994012 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 18:42:26.994116 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:42:26.994884 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:26.984453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:33523 (local address 127.18.235.129:45361)
0927 18:42:26.985004 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:26.985039 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:26.985767 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:26.986216 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:26.986222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:26.986515 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:26.986931 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:26.986944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:26.987676 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:26.987678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:26.988531 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:26.988541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:26.988944 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:26.988946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:26.989078 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:26.989574 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:26.989589 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:26.991050 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:26.992547 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:26.992553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:26.992566 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:26.992868 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:26.993149 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:26.993152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:26.993154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:26.993271 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:26.993622 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:26.993626 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:26.993811 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 18:42:26.993950 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:42:26.994155 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":94,"threads_started":1}
I20260927 18:42:26.995898 21995 heartbeater.cc:347] Connected to a master server at 127.18.235.190:33523
I20260927 18:42:26.996088 21995 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:26.996472 21995 heartbeater.cc:511] Master 127.18.235.190:33523 requested a full tablet report, sending...
I20260927 18:42:26.997795 21804 ts_manager.cc:194] Registered new tserver with Master: 04185c99c31244e0912adf534847a08a (127.18.235.129:36297)
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:42:26.998857 21804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:45361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:27.013958 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33523
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:27.114310 22000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:27.114511 22000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:27.114548 22000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:27.118377 22000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:27.118419 22000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:27.118472 22000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:27.123281 22000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:27.124050 22000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:27.124852 22000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:27.132944 22006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:27.132984 22008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:27.132961 22000 server_base.cc:1034] running on GCE node
W20260927 18:42:27.132987 22005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:27.133436 22000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:27.133929 22000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:27.135061 22000 hybrid_clock.cc:648] HybridClock initialized: now 1790534547135045 us; error 34 us; skew 500 ppm
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534547, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:27.137427 22000 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:27.138093 22000 webserver.cc:460] Webserver started at http://127.18.235.130:44961/ using document root <none> and password file <none>
I20260927 18:42:27.138528 22000 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:27.138579 22000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:27.138721 22000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:27.140413 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "25f4c9c5361b430c8bbdc82d620664a9"
format_stamp: "Formatted at 2026-09-27 18:42:27 on dist-test-slave-6lr6"
server_key: "54e16b76d838852983e7616c2acd44f3"
server_key_iv: "60eb242521238eb2010b731c93d1c718"
server_key_version: "encryptionkey@0"
I20260927 18:42:27.140856 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "25f4c9c5361b430c8bbdc82d620664a9"
format_stamp: "Formatted at 2026-09-27 18:42:27 on dist-test-slave-6lr6"
server_key: "54e16b76d838852983e7616c2acd44f3"
server_key_iv: "60eb242521238eb2010b731c93d1c718"
server_key_version: "encryptionkey@0"
I20260927 18:42:27.144057 22000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:27.146096 22015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:27.147130 22000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:27.147233 22000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "25f4c9c5361b430c8bbdc82d620664a9"
format_stamp: "Formatted at 2026-09-27 18:42:27 on dist-test-slave-6lr6"
server_key: "54e16b76d838852983e7616c2acd44f3"
server_key_iv: "60eb242521238eb2010b731c93d1c718"
server_key_version: "encryptionkey@0"
I20260927 18:42:27.147312 22000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:27.181061 22000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:27.183483 22000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:27.183635 22000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:27.184128 22000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:27.184846 22000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:27.184888 22000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:27.184937 22000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:27.184965 22000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:27.192957 22000 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43167
I20260927 18:42:27.192984 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43167 every 8 connection(s)
I20260927 18:42:27.193763 22000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:27.198576 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22000
I20260927 18:42:27.198657 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:27.198904 19374 external_mini_cluster.cc:1428] Setting key 7ecb415cf212af03a9cd4b4600e76ed9
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[21759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534547, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:27.206229 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:27.195324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:38811 (local address 127.18.235.190:33523)
0927 18:42:27.195455 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:27.195459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:27.196398 (+ 939us) server_negotiation.cc:407] Connection header received
0927 18:42:27.197255 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:27.197258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:27.197312 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:27.197383 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:27.198475 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.199047 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.200280 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.200493 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.200574 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:27.202862 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:27.202865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:27.202868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:27.202896 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:27.204361 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:27.204809 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:27.204812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:27.204814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:27.204901 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:27.205205 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:27.205209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:27.205211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:27.205425 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:27.205524 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:27.205935 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 18:42:27.206064 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:42:27.206844 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:27.195669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:33523 (local address 127.18.235.130:38811)
0927 18:42:27.196266 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:27.196303 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:27.197045 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:27.197549 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:27.197556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:27.197885 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:27.198328 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:27.198341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.199154 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.199158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:27.200175 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:27.200185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.200615 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.200619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:27.200787 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:27.201320 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:27.201334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:27.202712 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:27.204475 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:27.204480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:27.204490 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:27.204721 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:27.204986 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:27.204988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:27.204989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:27.205135 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:27.205511 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:27.205515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:27.205716 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 18:42:27.205850 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 18:42:27.206066 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:42:27.207828 22129 heartbeater.cc:347] Connected to a master server at 127.18.235.190:33523
I20260927 18:42:27.208024 22129 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:27.208494 22129 heartbeater.cc:511] Master 127.18.235.190:33523 requested a full tablet report, sending...
I20260927 18:42:27.209424 21804 ts_manager.cc:194] Registered new tserver with Master: 25f4c9c5361b430c8bbdc82d620664a9 (127.18.235.130:43167)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:42:27.210019 21804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:38811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:27.222124 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33523
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:27.323413 22134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:27.323635 22134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:27.323700 22134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:27.327539 22134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:27.327586 22134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:27.327665 22134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:27.332458 22134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:27.333343 22134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:27.334213 22134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:27.342718 22140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:27.342679 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:27.342728 22134 server_base.cc:1034] running on GCE node
W20260927 18:42:27.342679 22139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:27.343268 22134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:27.343827 22134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:27.344973 22134 hybrid_clock.cc:648] HybridClock initialized: now 1790534547344960 us; error 30 us; skew 500 ppm
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534547, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:27.347410 22134 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:27.348194 22134 webserver.cc:460] Webserver started at http://127.18.235.131:43819/ using document root <none> and password file <none>
I20260927 18:42:27.348656 22134 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:27.348717 22134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:27.348902 22134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:27.350776 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "6195919cc9d74a87b9b69cb2d640cc88"
format_stamp: "Formatted at 2026-09-27 18:42:27 on dist-test-slave-6lr6"
server_key: "862b7c7fec6d03b400d92dac64b36d46"
server_key_iv: "3fa878423824719d8d94d699e8c5515b"
server_key_version: "encryptionkey@0"
I20260927 18:42:27.351279 22134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "6195919cc9d74a87b9b69cb2d640cc88"
format_stamp: "Formatted at 2026-09-27 18:42:27 on dist-test-slave-6lr6"
server_key: "862b7c7fec6d03b400d92dac64b36d46"
server_key_iv: "3fa878423824719d8d94d699e8c5515b"
server_key_version: "encryptionkey@0"
I20260927 18:42:27.354573 22134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:27.356648 22149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:27.357736 22134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:27.357831 22134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "6195919cc9d74a87b9b69cb2d640cc88"
format_stamp: "Formatted at 2026-09-27 18:42:27 on dist-test-slave-6lr6"
server_key: "862b7c7fec6d03b400d92dac64b36d46"
server_key_iv: "3fa878423824719d8d94d699e8c5515b"
server_key_version: "encryptionkey@0"
I20260927 18:42:27.357913 22134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:27.370678 22134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:27.372915 22134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:27.373061 22134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:27.373559 22134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:27.374258 22134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:27.374301 22134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:27.374346 22134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:27.374375 22134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:27.382134 22134 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:36345
I20260927 18:42:27.382153 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:36345 every 8 connection(s)
I20260927 18:42:27.382859 22134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:27.386336 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22134
I20260927 18:42:27.386428 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:27.386660 19374 external_mini_cluster.cc:1428] Setting key ac015655c647299e2af307864e99476c
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[21759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534547, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:27.394342 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:27.384355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:52975 (local address 127.18.235.190:33523)
0927 18:42:27.384467 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:27.384471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:27.385364 (+ 893us) server_negotiation.cc:407] Connection header received
0927 18:42:27.386170 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:27.386173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:27.386235 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:27.386303 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:27.387445 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.388079 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.389247 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.389459 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.389543 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:27.391572 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:27.391575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:27.391577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:27.391600 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:27.392837 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:27.393261 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:27.393263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:27.393264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:27.393334 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:27.393567 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:27.393569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:27.393570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:27.393731 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:27.393797 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:27.394153 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 18:42:27.394236 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:42:27.394994 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:27.384660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:33523 (local address 127.18.235.131:52975)
0927 18:42:27.385248 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:27.385283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:27.385970 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:27.386472 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:27.386480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:27.386832 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:27.387316 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:27.387335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.388202 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.388205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:27.389137 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:27.389150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:27.389582 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:27.389586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:27.389735 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:27.390152 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:27.390166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:27.391429 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:27.392924 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:27.392928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:27.392938 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:27.393194 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:27.393395 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:27.393396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:27.393398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:27.393505 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:27.393798 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:27.393801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:27.393986 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 18:42:27.394112 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 18:42:27.394319 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":96,"threads_started":1}
I20260927 18:42:27.395931 22263 heartbeater.cc:347] Connected to a master server at 127.18.235.190:33523
I20260927 18:42:27.396119 22263 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:27.396523 22263 heartbeater.cc:511] Master 127.18.235.190:33523 requested a full tablet report, sending...
I20260927 18:42:27.397377 21804 ts_manager.cc:194] Registered new tserver with Master: 6195919cc9d74a87b9b69cb2d640cc88 (127.18.235.131:36345)
I20260927 18:42:27.397882 21804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:52975
I20260927 18:42:27.399890 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:42:27.410425 21843 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:27.409302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.18.235.190:33523)
0927 18:42:27.409426 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:27.409430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:27.409568 (+ 138us) server_negotiation.cc:407] Connection header received
0927 18:42:27.409657 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:27.409659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:27.409693 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:27.409753 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:27.410293 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47518: BlockingRecv error: recv got EOF from 127.0.0.1:47518 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:42:27.411345 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 21866
I20260927 18:42:27.416281 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22000
I20260927 18:42:27.420960 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22134
I20260927 18:42:27.425139 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 21772
2026-09-27T18:42:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1956 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[22275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[22275](info): set up 2 sockets
Sep 27 18:42:27 dist-test-slave-6lr6 krb5kdc[22275](info): commencing operation
krb5kdc: starting...
W20260927 18:42:28.364796 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:28 dist-test-slave-6lr6 krb5kdc[22275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:28.479507 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:37035
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:28.580204 22288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:28.580431 22288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:28.580482 22288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:28.584326 22288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:28.584372 22288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:28.584398 22288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:28.584419 22288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:28.584447 22288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:28.589300 22288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:37035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:28.590111 22288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:28.591061 22288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:28.598409 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:28.598417 22296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:28.598574 22288 server_base.cc:1034] running on GCE node
W20260927 18:42:28.598420 22294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:28.599249 22288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:28.600175 22288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:28.601315 22288 hybrid_clock.cc:648] HybridClock initialized: now 1790534548601301 us; error 37 us; skew 500 ppm
Sep 27 18:42:28 dist-test-slave-6lr6 krb5kdc[22275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534548, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:28.603619 22288 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:28.604375 22288 webserver.cc:460] Webserver started at http://127.18.235.190:44527/ using document root <none> and password file <none>
I20260927 18:42:28.604835 22288 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:28.604897 22288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:28.605072 22288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:28.606863 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "040feb73be464d488b92e4b31e8a40bd"
format_stamp: "Formatted at 2026-09-27 18:42:28 on dist-test-slave-6lr6"
server_key: "4341657909ccfb38a5e071c27651b533"
server_key_iv: "9bc57f9a1ade40b86ec1e89e1d2ec4e3"
server_key_version: "encryptionkey@0"
I20260927 18:42:28.607352 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "040feb73be464d488b92e4b31e8a40bd"
format_stamp: "Formatted at 2026-09-27 18:42:28 on dist-test-slave-6lr6"
server_key: "4341657909ccfb38a5e071c27651b533"
server_key_iv: "9bc57f9a1ade40b86ec1e89e1d2ec4e3"
server_key_version: "encryptionkey@0"
I20260927 18:42:28.610524 22288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 18:42:28.612548 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:28.613605 22288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:42:28.613708 22288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "040feb73be464d488b92e4b31e8a40bd"
format_stamp: "Formatted at 2026-09-27 18:42:28 on dist-test-slave-6lr6"
server_key: "4341657909ccfb38a5e071c27651b533"
server_key_iv: "9bc57f9a1ade40b86ec1e89e1d2ec4e3"
server_key_version: "encryptionkey@0"
I20260927 18:42:28.613799 22288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:28.622362 22288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:28.624706 22288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:28.624858 22288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:28.630694 22288 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:37035
I20260927 18:42:28.630918 22355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:37035 every 8 connection(s)
I20260927 18:42:28.631484 22288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:28.633484 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22288
I20260927 18:42:28.633582 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:28.633821 19374 external_mini_cluster.cc:1428] Setting key 696b4f5323e6d1128fca5be85c7b9f19
I20260927 18:42:28.633946 22356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:28.640306 22356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd: Bootstrap starting.
Sep 27 18:42:28 dist-test-slave-6lr6 krb5kdc[22275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:28.642508 22356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:28.643283 22356 log.cc:824] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:28.645195 22356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd: No bootstrap required, opened a new log
I20260927 18:42:28.647092 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:28.634882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.18.235.190:37035)
0927 18:42:28.635250 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:28.635267 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:28.635303 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:42:28.635966 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:28.635986 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:28.636354 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:28.636680 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:28.637340 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:28.638733 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:28.639510 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:28.640149 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:28.640294 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:28.642584 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:28.642590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:28.642604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:28.642633 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:28.644197 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:28.644572 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:28.644577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:28.644582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:28.644702 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:28.644932 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:28.644935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:28.644937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:28.645402 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:28.645577 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:28.645810 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 18:42:28.646073 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260927 18:42:28.648726 22356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040feb73be464d488b92e4b31e8a40bd" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37035 } }
I20260927 18:42:28.649001 22356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:28.649065 22356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040feb73be464d488b92e4b31e8a40bd, State: Initialized, Role: FOLLOWER
I20260927 18:42:28.649497 22356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040feb73be464d488b92e4b31e8a40bd" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37035 } }
I20260927 18:42:28.649658 22356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:28.649714 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:42:28.649736 22356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:28.649835 22356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:28.650835 22356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040feb73be464d488b92e4b31e8a40bd" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37035 } }
I20260927 18:42:28.651183 22356 leader_election.cc:302] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040feb73be464d488b92e4b31e8a40bd; no voters:
I20260927 18:42:28.651480 22356 leader_election.cc:288] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:28.651558 22361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:28.651834 22361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [term 1 LEADER]: Becoming Leader. State: Replica: 040feb73be464d488b92e4b31e8a40bd, State: Running, Role: LEADER
I20260927 18:42:28.652161 22361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040feb73be464d488b92e4b31e8a40bd" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37035 } }
I20260927 18:42:28.652613 22356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:28.653609 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "040feb73be464d488b92e4b31e8a40bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040feb73be464d488b92e4b31e8a40bd" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37035 } } }
I20260927 18:42:28.653800 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:28.654031 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040feb73be464d488b92e4b31e8a40bd. Latest consensus state: current_term: 1 leader_uuid: "040feb73be464d488b92e4b31e8a40bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040feb73be464d488b92e4b31e8a40bd" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37035 } } }
I20260927 18:42:28.654172 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd [sys.catalog]: This master's current role is: LEADER
W20260927 18:42:28.657228 22378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:28.657294 22378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:28.657382 22371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:28.658140 22371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:28.663223 22371 catalog_manager.cc:1347] Generated new cluster ID: 6b8a363cf7954e7792bd6823b31176a7
I20260927 18:42:28.663276 22371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:28.667563 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:28.662304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.18.235.190:37035)
0927 18:42:28.662425 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:28.662430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:28.662659 (+ 229us) server_negotiation.cc:407] Connection header received
0927 18:42:28.662783 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:28.662786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:28.662830 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:28.662892 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:28.663488 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:28.664265 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:28.665433 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:28.665776 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:28.665880 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:28.666680 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:28.666684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:28.666686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:28.667146 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:28.667176 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:28.667195 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:28.667288 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:28.667385 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 18:42:28.667438 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:42:28.679188 22371 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:28.679968 22371 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:28.688546 22371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040feb73be464d488b92e4b31e8a40bd: Generated new TSK 0
I20260927 18:42:28.689244 22371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:42:28.701840 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22288
2026-09-27T18:42:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1274 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:28 dist-test-slave-6lr6 krb5kdc[22389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:28 dist-test-slave-6lr6 krb5kdc[22389](info): set up 2 sockets
Sep 27 18:42:28 dist-test-slave-6lr6 krb5kdc[22389](info): commencing operation
krb5kdc: starting...
W20260927 18:42:29.636637 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:29 dist-test-slave-6lr6 krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:29.739765 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45423
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:45423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:29.852125 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:29.852340 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:29.852392 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:29.856245 22402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:29.856293 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:29.856317 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:29.856339 22402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:29.856367 22402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:29.861218 22402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:45423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:29.862073 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:29.862963 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:29.870321 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:29.870364 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:29.870512 22402 server_base.cc:1034] running on GCE node
W20260927 18:42:29.870364 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:29.871083 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:29.872017 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:29.873164 22402 hybrid_clock.cc:648] HybridClock initialized: now 1790534549873146 us; error 28 us; skew 500 ppm
Sep 27 18:42:29 dist-test-slave-6lr6 krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534549, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:29.875576 22402 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:29.876329 22402 webserver.cc:460] Webserver started at http://127.18.235.190:38909/ using document root <none> and password file <none>
I20260927 18:42:29.876811 22402 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:29.876869 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:29.877032 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:29.878856 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "560310c61fb1493fb9511a3faa41c963"
format_stamp: "Formatted at 2026-09-27 18:42:29 on dist-test-slave-6lr6"
server_key: "413b590d99813ed32eb3f45ecb8adb12"
server_key_iv: "2e7fa53ca23a60875f97dddda39c3809"
server_key_version: "encryptionkey@0"
I20260927 18:42:29.879354 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "560310c61fb1493fb9511a3faa41c963"
format_stamp: "Formatted at 2026-09-27 18:42:29 on dist-test-slave-6lr6"
server_key: "413b590d99813ed32eb3f45ecb8adb12"
server_key_iv: "2e7fa53ca23a60875f97dddda39c3809"
server_key_version: "encryptionkey@0"
I20260927 18:42:29.882594 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:29.884666 22417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:29.885798 22402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:29.885897 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "560310c61fb1493fb9511a3faa41c963"
format_stamp: "Formatted at 2026-09-27 18:42:29 on dist-test-slave-6lr6"
server_key: "413b590d99813ed32eb3f45ecb8adb12"
server_key_iv: "2e7fa53ca23a60875f97dddda39c3809"
server_key_version: "encryptionkey@0"
I20260927 18:42:29.885985 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:29.921663 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:29.924079 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:29.924232 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:29.930471 22402 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:45423
I20260927 18:42:29.930706 22469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:45423 every 8 connection(s)
I20260927 18:42:29.931308 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:29.933785 22470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:29.934178 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22402
I20260927 18:42:29.934305 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:29.934589 19374 external_mini_cluster.cc:1428] Setting key 6b117327b3ab14f90499de74e1a0f138
I20260927 18:42:29.940073 22470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963: Bootstrap starting.
I20260927 18:42:29.942528 22470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963: Neither blocks nor log segments found. Creating new log.
Sep 27 18:42:29 dist-test-slave-6lr6 krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:29.943189 22470 log.cc:824] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:29.945255 22470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963: No bootstrap required, opened a new log
I20260927 18:42:29.947839 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:29.935785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.18.235.190:45423)
0927 18:42:29.936162 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:29.936176 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:29.936211 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:42:29.936850 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:29.936865 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:29.937188 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:29.937445 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:29.938232 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:29.939156 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:29.940277 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:29.940713 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:29.940819 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:29.943309 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:29.943315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:29.943329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:29.943358 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:29.945126 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:29.945515 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:29.945520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:29.945525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:29.945657 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:29.945867 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:29.945870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:29.945872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:29.946329 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:29.946498 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:29.946668 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:42:29.946935 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20260927 18:42:29.949087 22470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560310c61fb1493fb9511a3faa41c963" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45423 } }
I20260927 18:42:29.949319 22470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:29.949365 22470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560310c61fb1493fb9511a3faa41c963, State: Initialized, Role: FOLLOWER
I20260927 18:42:29.949764 22470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560310c61fb1493fb9511a3faa41c963" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45423 } }
I20260927 18:42:29.949875 22470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:29.949929 22470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:29.950017 22470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:29.950753 22470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560310c61fb1493fb9511a3faa41c963" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45423 } }
I20260927 18:42:29.951040 22470 leader_election.cc:302] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560310c61fb1493fb9511a3faa41c963; no voters:
I20260927 18:42:29.951277 22470 leader_election.cc:288] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:29.951373 22475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:29.951557 22475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [term 1 LEADER]: Becoming Leader. State: Replica: 560310c61fb1493fb9511a3faa41c963, State: Running, Role: LEADER
I20260927 18:42:29.951858 22475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560310c61fb1493fb9511a3faa41c963" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45423 } }
I20260927 18:42:29.952283 22470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:29.953236 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560310c61fb1493fb9511a3faa41c963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560310c61fb1493fb9511a3faa41c963" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45423 } } }
I20260927 18:42:29.953351 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:29.953675 22484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:29.953647 22477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560310c61fb1493fb9511a3faa41c963. Latest consensus state: current_term: 1 leader_uuid: "560310c61fb1493fb9511a3faa41c963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560310c61fb1493fb9511a3faa41c963" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45423 } } }
I20260927 18:42:29.954284 22477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:29.956512 22484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:29.961541 22484 catalog_manager.cc:1347] Generated new cluster ID: d2fda08997dc4b8883b59ba8337f09ba
I20260927 18:42:29.961596 22484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:29.983469 22484 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:29.984562 22484 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:29.989533 22484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560310c61fb1493fb9511a3faa41c963: Generated new TSK 0
I20260927 18:42:29.990253 22484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:30.016134 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45423
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:30.118460 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:30.118674 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:30.118721 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:30.122641 22496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:30.122686 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:30.122737 22496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:30.127614 22496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:30.128415 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:30.129268 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:30.137885 22501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:30.137933 22504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:30.137915 22496 server_base.cc:1034] running on GCE node
W20260927 18:42:30.138067 22502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:30.138391 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:30.138911 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:30.140053 22496 hybrid_clock.cc:648] HybridClock initialized: now 1790534550140039 us; error 27 us; skew 500 ppm
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:30.142482 22496 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:30.143195 22496 webserver.cc:460] Webserver started at http://127.18.235.129:35267/ using document root <none> and password file <none>
I20260927 18:42:30.143652 22496 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:30.143705 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:30.143887 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:30.145704 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "a9e7c59855d240e6abda7d6aa96e38e8"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "b903e5a8ba9d243428ad804a505fd643"
server_key_iv: "db1713a14012154f52860dbbc054d26a"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.146196 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "a9e7c59855d240e6abda7d6aa96e38e8"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "b903e5a8ba9d243428ad804a505fd643"
server_key_iv: "db1713a14012154f52860dbbc054d26a"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.149376 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:42:30.151425 22511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.152523 22496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:30.152613 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "a9e7c59855d240e6abda7d6aa96e38e8"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "b903e5a8ba9d243428ad804a505fd643"
server_key_iv: "db1713a14012154f52860dbbc054d26a"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.152695 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:30.167740 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:30.170426 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:30.170573 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:30.171051 22496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:30.171726 22496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:30.171768 22496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.171803 22496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:30.171839 22496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.179514 22496 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:33895
I20260927 18:42:30.179569 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:33895 every 8 connection(s)
I20260927 18:42:30.180253 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:30.180894 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22496
I20260927 18:42:30.180995 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:30.181300 19374 external_mini_cluster.cc:1428] Setting key 9329cf8290b70e1e0287aa607a75fc69
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:42:30.193524 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.181788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:40919 (local address 127.18.235.190:45423)
0927 18:42:30.181921 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:30.181926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:30.182775 (+ 849us) server_negotiation.cc:407] Connection header received
0927 18:42:30.184306 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:30.184310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:30.184361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:30.184437 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:30.185426 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.185953 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.187535 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.187768 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.187845 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.190440 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:30.190444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:30.190446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:30.190470 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:30.191676 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.192123 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.192126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.192128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.192207 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.192521 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.192525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.192526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.192712 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:30.192799 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:30.193303 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 18:42:30.193406 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:42:30.194502 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.182160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45423 (local address 127.18.235.129:40919)
0927 18:42:30.182625 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:30.182675 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:30.184087 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:30.184562 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:30.184568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:30.184870 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:30.185307 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:30.185319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.186075 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.186079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:30.187416 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:30.187428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.187908 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.187912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:30.188105 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.188594 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:30.188614 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:30.190291 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:30.191787 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:30.191793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:30.191807 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:30.192036 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:30.192316 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:30.192318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:30.192320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:30.192438 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:30.192797 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:30.192804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:30.193039 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 18:42:30.193256 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 18:42:30.193562 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":89,"threads_started":1}
I20260927 18:42:30.195619 22625 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45423
I20260927 18:42:30.195835 22625 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:30.196265 22625 heartbeater.cc:511] Master 127.18.235.190:45423 requested a full tablet report, sending...
I20260927 18:42:30.197793 22434 ts_manager.cc:194] Registered new tserver with Master: a9e7c59855d240e6abda7d6aa96e38e8 (127.18.235.129:33895)
I20260927 18:42:30.199345 22434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:40919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:30.207197 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45423
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:30.307384 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:30.307587 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:30.307626 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:30.311465 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:30.311506 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:30.311559 22630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:30.316337 22630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:30.317124 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:30.317929 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:30.326107 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:30.326187 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:30.326342 22630 server_base.cc:1034] running on GCE node
W20260927 18:42:30.326113 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:30.326736 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:30.327284 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:30.328442 22630 hybrid_clock.cc:648] HybridClock initialized: now 1790534550328430 us; error 24 us; skew 500 ppm
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:30.330744 22630 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:30.331504 22630 webserver.cc:460] Webserver started at http://127.18.235.130:36631/ using document root <none> and password file <none>
I20260927 18:42:30.331957 22630 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:30.332015 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:30.332190 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:30.333973 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "d6141f54fa49454ea8dfd4883971499d"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "7b4184df3b0076cffe0215f0e459b6dd"
server_key_iv: "aa033961244f89773e0178c5596ff6c3"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.334475 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "d6141f54fa49454ea8dfd4883971499d"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "7b4184df3b0076cffe0215f0e459b6dd"
server_key_iv: "aa033961244f89773e0178c5596ff6c3"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.337616 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:30.339646 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.340660 22630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:30.340752 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "d6141f54fa49454ea8dfd4883971499d"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "7b4184df3b0076cffe0215f0e459b6dd"
server_key_iv: "aa033961244f89773e0178c5596ff6c3"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.340830 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:30.359861 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:30.362118 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:30.362262 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:30.362746 22630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:30.363422 22630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:30.363471 22630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.363518 22630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:30.363545 22630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.371069 22630 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:37379
I20260927 18:42:30.371094 22758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:37379 every 8 connection(s)
I20260927 18:42:30.371784 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:30.381976 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22630
I20260927 18:42:30.382084 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:30.382390 19374 external_mini_cluster.cc:1428] Setting key 516baef5112a5ce5d4283fdace739cf7
I20260927 18:42:30.383263 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.373288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:35439 (local address 127.18.235.190:45423)
0927 18:42:30.373414 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:30.373418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:30.374153 (+ 735us) server_negotiation.cc:407] Connection header received
0927 18:42:30.375314 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:30.375317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:30.375362 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:30.375443 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:30.376404 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.376907 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.377979 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.378197 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.378272 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.380228 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:30.380231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:30.380233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:30.380254 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:30.381474 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.381895 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.381898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.381899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.381964 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.382284 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.382286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.382287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.382455 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:30.382547 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:30.382987 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 18:42:30.383106 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:42:30.383970 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.373639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45423 (local address 127.18.235.130:35439)
0927 18:42:30.374020 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:30.374055 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:30.375143 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:30.375569 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:30.375574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:30.375895 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:30.376302 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:30.376314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.376987 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.376989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:30.377867 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:30.377876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.378299 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.378301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:30.378421 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.378851 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:30.378866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:30.380094 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:30.381563 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:30.381567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:30.381576 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:30.381819 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:30.382038 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:30.382040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:30.382042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:30.382198 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:30.382557 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:30.382562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:30.382766 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 18:42:30.382932 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 18:42:30.383173 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260927 18:42:30.384935 22759 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45423
I20260927 18:42:30.385147 22759 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:30.385628 22759 heartbeater.cc:511] Master 127.18.235.190:45423 requested a full tablet report, sending...
I20260927 18:42:30.386596 22434 ts_manager.cc:194] Registered new tserver with Master: d6141f54fa49454ea8dfd4883971499d (127.18.235.130:37379)
I20260927 18:42:30.387113 22434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:35439
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:30.406242 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45423
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:30.506040 22764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:30.506253 22764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:30.506297 22764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:30.510123 22764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:30.510169 22764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:30.510234 22764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:30.515146 22764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:30.516026 22764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:30.516927 22764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:30.525336 22772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:30.525313 22769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:30.525365 22764 server_base.cc:1034] running on GCE node
W20260927 18:42:30.525311 22770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:30.525909 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:30.526446 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:30.527602 22764 hybrid_clock.cc:648] HybridClock initialized: now 1790534550527584 us; error 31 us; skew 500 ppm
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:30.535930 22764 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:30.536664 22764 webserver.cc:460] Webserver started at http://127.18.235.131:34717/ using document root <none> and password file <none>
I20260927 18:42:30.537174 22764 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:30.537235 22764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:30.537425 22764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:30.539229 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "2b7937ef21f3453185c1e6c01c9a2892"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "6dd49a4a425aacc55cbefc0622d71f3e"
server_key_iv: "0ee0d2036baa4e3342ffbb860daecb09"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.539723 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "2b7937ef21f3453185c1e6c01c9a2892"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "6dd49a4a425aacc55cbefc0622d71f3e"
server_key_iv: "0ee0d2036baa4e3342ffbb860daecb09"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.542999 22764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:30.545408 22779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.546468 22764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:30.546554 22764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "2b7937ef21f3453185c1e6c01c9a2892"
format_stamp: "Formatted at 2026-09-27 18:42:30 on dist-test-slave-6lr6"
server_key: "6dd49a4a425aacc55cbefc0622d71f3e"
server_key_iv: "0ee0d2036baa4e3342ffbb860daecb09"
server_key_version: "encryptionkey@0"
I20260927 18:42:30.546644 22764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:30.562292 22764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:30.564544 22764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:30.564682 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:30.565176 22764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:30.565860 22764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:30.565900 22764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.565937 22764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:30.565966 22764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:30.573421 22764 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:46701
I20260927 18:42:30.573469 22892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:46701 every 8 connection(s)
I20260927 18:42:30.574146 22764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:30.580575 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22764
I20260927 18:42:30.580673 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:30.580935 19374 external_mini_cluster.cc:1428] Setting key 47feb060687086ef7694d62c08fd3514
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:30.585453 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.575602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:59781 (local address 127.18.235.190:45423)
0927 18:42:30.575710 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:30.575714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:30.576379 (+ 665us) server_negotiation.cc:407] Connection header received
0927 18:42:30.577587 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:30.577590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:30.577636 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:30.577714 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:30.578631 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.579134 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.580104 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.580306 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.580371 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.582627 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:30.582630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:30.582632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:30.582657 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:30.583737 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.584157 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.584160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.584162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.584233 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.584512 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.584515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.584517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.584671 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:30.584758 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:30.585194 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:42:30.585307 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:42:30.586071 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.575880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45423 (local address 127.18.235.131:59781)
0927 18:42:30.576246 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:30.576280 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:30.577421 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:30.577837 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:30.577843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:30.578133 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:30.578531 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:30.578543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.579215 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.579217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:30.580013 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:30.580021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.580425 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.580430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:30.580607 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.581074 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:30.581093 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:30.582488 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:30.583835 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:30.583840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:30.583853 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:30.584073 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:30.584310 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:30.584313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:30.584315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:30.584433 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:30.584769 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:30.584775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:30.584972 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 18:42:30.585138 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 18:42:30.585412 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260927 18:42:30.586915 22893 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45423
I20260927 18:42:30.587078 22893 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:30.587430 22893 heartbeater.cc:511] Master 127.18.235.190:45423 requested a full tablet report, sending...
I20260927 18:42:30.588215 22434 ts_manager.cc:194] Registered new tserver with Master: 2b7937ef21f3453185c1e6c01c9a2892 (127.18.235.131:46701)
I20260927 18:42:30.588727 22434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:59781
I20260927 18:42:30.594130 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534550, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:30.615557 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:30.608544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.18.235.190:45423)
0927 18:42:30.608648 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:30.608652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:30.608810 (+ 158us) server_negotiation.cc:407] Connection header received
0927 18:42:30.608930 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:30.608932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:30.608967 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:30.609023 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:30.609666 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.610144 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.610916 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:30.611118 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:30.611196 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:30.613176 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:30.613179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:30.613180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:30.613200 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:30.614173 (+ 973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.614518 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.614522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.614524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.614587 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:30.614807 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:30.614810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:30.614812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:30.614993 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:30.615102 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:30.615312 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:42:30.615424 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 18:42:30.615842 22434 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38766
I20260927 18:42:30.616604 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22496
I20260927 18:42:30.621239 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22630
I20260927 18:42:30.625443 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22764
I20260927 18:42:30.629535 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22402
2026-09-27T18:42:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1928 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22905](info): set up 2 sockets
Sep 27 18:42:30 dist-test-slave-6lr6 krb5kdc[22905](info): commencing operation
krb5kdc: starting...
W20260927 18:42:31.576638 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:31 dist-test-slave-6lr6 krb5kdc[22905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:31.679939 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32803
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:31.780118 22918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:31.780344 22918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:31.780395 22918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:31.784241 22918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:31.784288 22918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:31.784314 22918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:31.784334 22918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:31.784355 22918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:31.789290 22918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:32803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:31.790174 22918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:31.791062 22918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:31.798653 22924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:31.798693 22923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:31.798738 22918 server_base.cc:1034] running on GCE node
W20260927 18:42:31.798693 22926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:31.799342 22918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:31.800292 22918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:31.801435 22918 hybrid_clock.cc:648] HybridClock initialized: now 1790534551801423 us; error 24 us; skew 500 ppm
Sep 27 18:42:31 dist-test-slave-6lr6 krb5kdc[22905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534551, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:31.803958 22918 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:31.804729 22918 webserver.cc:460] Webserver started at http://127.18.235.190:45649/ using document root <none> and password file <none>
I20260927 18:42:31.805255 22918 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:31.805315 22918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:31.805501 22918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:31.807335 22918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "a99e2c5477734976965fbbd21971238e"
format_stamp: "Formatted at 2026-09-27 18:42:31 on dist-test-slave-6lr6"
server_key: "31f59de8c1f84342190013c8bbc6b0ca"
server_key_iv: "ce3a9d4ed547879e127a402600b14192"
server_key_version: "encryptionkey@0"
I20260927 18:42:31.807837 22918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "a99e2c5477734976965fbbd21971238e"
format_stamp: "Formatted at 2026-09-27 18:42:31 on dist-test-slave-6lr6"
server_key: "31f59de8c1f84342190013c8bbc6b0ca"
server_key_iv: "ce3a9d4ed547879e127a402600b14192"
server_key_version: "encryptionkey@0"
I20260927 18:42:31.811131 22918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:31.813235 22933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:31.814309 22918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:31.814411 22918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "a99e2c5477734976965fbbd21971238e"
format_stamp: "Formatted at 2026-09-27 18:42:31 on dist-test-slave-6lr6"
server_key: "31f59de8c1f84342190013c8bbc6b0ca"
server_key_iv: "ce3a9d4ed547879e127a402600b14192"
server_key_version: "encryptionkey@0"
I20260927 18:42:31.814492 22918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:31.831334 22918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:31.833916 22918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:31.834057 22918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:31.840081 22918 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:32803
I20260927 18:42:31.840093 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:32803 every 8 connection(s)
I20260927 18:42:31.840874 22918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:31.843287 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:31.844133 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 22918
I20260927 18:42:31.844231 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:31.844465 19374 external_mini_cluster.cc:1428] Setting key 1bdfb7c2ebd26968332a39e291ec9ae0
I20260927 18:42:31.848523 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e: Bootstrap starting.
I20260927 18:42:31.850488 22986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:31.851101 22986 log.cc:824] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:42:31 dist-test-slave-6lr6 krb5kdc[22905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:31.853003 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e: No bootstrap required, opened a new log
I20260927 18:42:31.856541 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:31.845642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.18.235.190:32803)
0927 18:42:31.846025 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:31.846038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:31.846069 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:42:31.846643 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:31.846657 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:31.846953 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:31.847224 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:31.847966 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:31.848902 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:31.849727 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:31.850168 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:31.850308 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:31.852714 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:31.852719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:31.852731 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:31.852754 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:31.854188 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:31.854624 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:31.854628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:31.854632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:31.854722 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:31.854968 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:31.854970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:31.854971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:31.855276 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:31.855411 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:31.855670 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 18:42:31.855900 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260927 18:42:31.856552 22986 raft_consensus.cc:348] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e2c5477734976965fbbd21971238e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32803 } }
I20260927 18:42:31.856832 22986 raft_consensus.cc:374] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:31.856892 22986 raft_consensus.cc:729] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99e2c5477734976965fbbd21971238e, State: Initialized, Role: FOLLOWER
I20260927 18:42:31.857286 22986 consensus_queue.cc:260] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e2c5477734976965fbbd21971238e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32803 } }
I20260927 18:42:31.857401 22986 raft_consensus.cc:388] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:31.857458 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:31.857542 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:31.858444 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e2c5477734976965fbbd21971238e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32803 } }
I20260927 18:42:31.858752 22986 leader_election.cc:302] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a99e2c5477734976965fbbd21971238e; no voters:
I20260927 18:42:31.858992 22986 leader_election.cc:288] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:31.859056 22991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:31.859236 22991 raft_consensus.cc:686] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [term 1 LEADER]: Becoming Leader. State: Replica: a99e2c5477734976965fbbd21971238e, State: Running, Role: LEADER
I20260927 18:42:31.859536 22991 consensus_queue.cc:237] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e2c5477734976965fbbd21971238e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32803 } }
I20260927 18:42:31.860026 22986 sys_catalog.cc:564] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:31.860994 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a99e2c5477734976965fbbd21971238e. Latest consensus state: current_term: 1 leader_uuid: "a99e2c5477734976965fbbd21971238e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e2c5477734976965fbbd21971238e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32803 } } }
I20260927 18:42:31.860996 22992 sys_catalog.cc:455] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a99e2c5477734976965fbbd21971238e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e2c5477734976965fbbd21971238e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32803 } } }
I20260927 18:42:31.861251 22992 sys_catalog.cc:458] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:31.861246 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:31.861896 23000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:31.864324 23000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:31.869067 23000 catalog_manager.cc:1347] Generated new cluster ID: f1a0744879844ec6ae8239c9e87fe8c1
I20260927 18:42:31.869143 23000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:31.886422 23000 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:31.887477 23000 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:31.899001 23000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a99e2c5477734976965fbbd21971238e: Generated new TSK 0
I20260927 18:42:31.899537 23000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:31.923604 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32803
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:32.027215 23012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:32.027474 23012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:32.027518 23012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:32.031514 23012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:32.031565 23012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:32.031647 23012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:32.036655 23012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:32.037618 23012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:32.038542 23012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:42:32.047188 23012 server_base.cc:1034] running on GCE node
W20260927 18:42:32.047140 23018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.047130 23017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.047132 23020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:32.047776 23012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:32.048347 23012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:32.049537 23012 hybrid_clock.cc:648] HybridClock initialized: now 1790534552049508 us; error 29 us; skew 500 ppm
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:32.051862 23012 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:32.052588 23012 webserver.cc:460] Webserver started at http://127.18.235.129:33819/ using document root <none> and password file <none>
I20260927 18:42:32.053061 23012 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:32.053150 23012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:32.053334 23012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:32.055104 23012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "e79ae43acd5e46b2a0dd195ef374b3f8"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "3555f67a225389db16fb4348771f27ce"
server_key_iv: "78121a7531da4dc4d958b6ba48290c13"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.055606 23012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "e79ae43acd5e46b2a0dd195ef374b3f8"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "3555f67a225389db16fb4348771f27ce"
server_key_iv: "78121a7531da4dc4d958b6ba48290c13"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.058853 23012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:32.060896 23027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.062048 23012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:32.062141 23012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "e79ae43acd5e46b2a0dd195ef374b3f8"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "3555f67a225389db16fb4348771f27ce"
server_key_iv: "78121a7531da4dc4d958b6ba48290c13"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.062227 23012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:32.085352 23012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:32.087571 23012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:32.087709 23012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:32.088203 23012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:32.088977 23012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:32.089021 23012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.089066 23012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:32.089094 23012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.097090 23012 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:41411
I20260927 18:42:32.097132 23140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:41411 every 8 connection(s)
I20260927 18:42:32.097895 23012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:32.098317 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23012
I20260927 18:42:32.098434 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:32.098693 19374 external_mini_cluster.cc:1428] Setting key 1f7fdc500879a3f13cd169625d350de4
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:42:32.110163 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.099400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:53025 (local address 127.18.235.190:32803)
0927 18:42:32.099519 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:32.099524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:32.100338 (+ 814us) server_negotiation.cc:407] Connection header received
0927 18:42:32.101658 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:32.101661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:32.101713 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:32.101784 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:32.102797 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.103363 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.104449 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.104670 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.104752 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.107090 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:32.107093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:32.107096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:32.107116 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:32.108380 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.108805 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.108808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.108809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.108878 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.109199 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.109202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.109204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.109374 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:32.109455 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:32.109889 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 18:42:32.110012 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:42:32.110802 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.099784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32803 (local address 127.18.235.129:53025)
0927 18:42:32.100195 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:32.100242 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:32.101477 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:32.101941 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:32.101947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:32.102250 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:32.102663 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.102675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.103474 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.103477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.104339 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.104349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.104796 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.104800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.104949 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.105460 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:32.105475 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:32.106954 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:32.108479 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.108484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.108496 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.108717 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.108952 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.108955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.108957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.109123 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.109463 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:32.109469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:32.109673 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 18:42:32.109813 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:42:32.110094 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
I20260927 18:42:32.111862 23141 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32803
I20260927 18:42:32.112061 23141 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:32.112465 23141 heartbeater.cc:511] Master 127.18.235.190:32803 requested a full tablet report, sending...
I20260927 18:42:32.113775 22950 ts_manager.cc:194] Registered new tserver with Master: e79ae43acd5e46b2a0dd195ef374b3f8 (127.18.235.129:41411)
I20260927 18:42:32.114792 22950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:53025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:32.121942 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32803
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:32.222295 23146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:32.222505 23146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:32.222541 23146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:32.226373 23146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:32.226416 23146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:32.226467 23146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:32.231303 23146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:32.232074 23146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:32.232882 23146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:32.241148 23151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.241137 23152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:32.241209 23146 server_base.cc:1034] running on GCE node
W20260927 18:42:32.241163 23154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:32.241719 23146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:32.242390 23146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:32.243564 23146 hybrid_clock.cc:648] HybridClock initialized: now 1790534552243536 us; error 41 us; skew 500 ppm
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:32.246054 23146 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:32.246815 23146 webserver.cc:460] Webserver started at http://127.18.235.130:37339/ using document root <none> and password file <none>
I20260927 18:42:32.247282 23146 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:32.247342 23146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:32.247509 23146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:32.249377 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "62738cbf086e4aeba143be80bf20d7c8"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "4ebcbf1922e762fc3e5fbc38f5fca1f6"
server_key_iv: "52111bc35b7a4be9e0b6f58f7fc45118"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.249876 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "62738cbf086e4aeba143be80bf20d7c8"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "4ebcbf1922e762fc3e5fbc38f5fca1f6"
server_key_iv: "52111bc35b7a4be9e0b6f58f7fc45118"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.253098 23146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:42:32.255188 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.256281 23146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:32.256382 23146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "62738cbf086e4aeba143be80bf20d7c8"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "4ebcbf1922e762fc3e5fbc38f5fca1f6"
server_key_iv: "52111bc35b7a4be9e0b6f58f7fc45118"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.256462 23146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:32.282092 23146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:32.284360 23146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:32.284512 23146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:32.284998 23146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:32.285727 23146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:32.285771 23146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.285820 23146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:32.285846 23146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.293505 23146 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:46483
I20260927 18:42:32.293529 23274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:46483 every 8 connection(s)
I20260927 18:42:32.294247 23146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:32.296633 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23146
I20260927 18:42:32.296752 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:32.297025 19374 external_mini_cluster.cc:1428] Setting key 6496953308cd48d614759612dfd68bdc
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:42:32.306882 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.295872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:46095 (local address 127.18.235.190:32803)
0927 18:42:32.296023 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:32.296028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:32.296775 (+ 747us) server_negotiation.cc:407] Connection header received
0927 18:42:32.298063 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:32.298066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:32.298119 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:32.298187 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:32.299218 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.299735 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.300812 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.301028 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.301135 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.303672 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:32.303675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:32.303694 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:32.303717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:32.304990 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.305549 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.305551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.305553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.305628 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.305955 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.305958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.305959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.306138 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:32.306240 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:32.306663 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:42:32.306773 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:42:32.307543 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.296186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32803 (local address 127.18.235.130:46095)
0927 18:42:32.296592 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:32.296629 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:32.297877 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:32.298360 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:32.298367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:32.298667 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:32.299086 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.299099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.299849 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.299851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.300691 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.300700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.301190 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.301194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.301343 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.301901 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:32.301916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:32.303518 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:32.305100 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.305127 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.305139 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.305455 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.305712 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.305714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.305716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.305836 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.306246 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:32.306252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:32.306452 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 18:42:32.306590 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:42:32.306833 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:42:32.308619 23275 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32803
I20260927 18:42:32.308817 23275 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:32.309278 23275 heartbeater.cc:511] Master 127.18.235.190:32803 requested a full tablet report, sending...
I20260927 18:42:32.310231 22950 ts_manager.cc:194] Registered new tserver with Master: 62738cbf086e4aeba143be80bf20d7c8 (127.18.235.130:46483)
I20260927 18:42:32.310777 22950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:46095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:32.321228 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32803
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:32.423378 23280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:32.423586 23280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:32.423640 23280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:32.427649 23280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:32.427698 23280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:32.427771 23280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:32.432969 23280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:32.433821 23280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:32.434681 23280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:32.442976 23285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.443020 23286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.442968 23288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:32.443159 23280 server_base.cc:1034] running on GCE node
I20260927 18:42:32.443524 23280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:32.444078 23280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:32.445240 23280 hybrid_clock.cc:648] HybridClock initialized: now 1790534552445210 us; error 40 us; skew 500 ppm
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:32.447582 23280 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:32.448328 23280 webserver.cc:460] Webserver started at http://127.18.235.131:33315/ using document root <none> and password file <none>
I20260927 18:42:32.448802 23280 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:32.448863 23280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:32.449041 23280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:32.450892 23280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "450fc3409fd148239ee6de80cff5897b"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "32839184803778951f16b8eca0d1c8f6"
server_key_iv: "207608ceb4367a3d27b57dc3141371ce"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.451381 23280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "450fc3409fd148239ee6de80cff5897b"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "32839184803778951f16b8eca0d1c8f6"
server_key_iv: "207608ceb4367a3d27b57dc3141371ce"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.454651 23280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 18:42:32.456736 23295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.457791 23280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:32.457894 23280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "450fc3409fd148239ee6de80cff5897b"
format_stamp: "Formatted at 2026-09-27 18:42:32 on dist-test-slave-6lr6"
server_key: "32839184803778951f16b8eca0d1c8f6"
server_key_iv: "207608ceb4367a3d27b57dc3141371ce"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.457973 23280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:32.471434 23280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:32.473681 23280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:32.473821 23280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:32.474287 23280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:32.474962 23280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:32.475005 23280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.475049 23280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:32.475077 23280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.482527 23280 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:46075
I20260927 18:42:32.482554 23408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:46075 every 8 connection(s)
I20260927 18:42:32.483289 23280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:32.485397 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23280
I20260927 18:42:32.485510 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:32.485790 19374 external_mini_cluster.cc:1428] Setting key 18a9bbaeaa1d52bf353c92c68afbe2dc
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:32.494800 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.484817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:54583 (local address 127.18.235.190:32803)
0927 18:42:32.484932 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:32.484936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:32.485743 (+ 807us) server_negotiation.cc:407] Connection header received
0927 18:42:32.487199 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:32.487201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:32.487244 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:32.487300 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:32.488391 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.488895 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.489944 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.490147 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.490210 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.492160 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:32.492163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:32.492165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:32.492186 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:32.493404 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.493777 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.493780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.493781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.493844 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.494068 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.494069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.494070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.494210 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:32.494274 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:32.494616 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 18:42:32.494694 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:42:32.495468 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.485140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32803 (local address 127.18.235.131:54583)
0927 18:42:32.485586 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:32.485637 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:32.487016 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:32.487467 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:32.487475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:32.487862 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:32.488276 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.488288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.488978 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.488980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.489856 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.489864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.490239 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.490242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.490363 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.490792 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:32.490807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:32.492031 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:32.493478 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.493482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.493490 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.493695 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.493901 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.493903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.493904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.494009 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.494271 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:32.494275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:32.494454 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 18:42:32.494584 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 18:42:32.494796 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260927 18:42:32.496521 23409 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32803
I20260927 18:42:32.496721 23409 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:32.497128 23409 heartbeater.cc:511] Master 127.18.235.190:32803 requested a full tablet report, sending...
I20260927 18:42:32.498011 22950 ts_manager.cc:194] Registered new tserver with Master: 450fc3409fd148239ee6de80cff5897b (127.18.235.131:46075)
I20260927 18:42:32.498515 22950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:54583
I20260927 18:42:32.499243 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:32.521304 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.513629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.18.235.129:41411)
0927 18:42:32.513939 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:32.513947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:32.513965 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:42:32.514006 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:32.514009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:32.514149 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:32.514258 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:32.514794 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.515525 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.516267 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.516585 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.516659 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.518677 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:32.518684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:32.518694 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:32.518714 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:32.519927 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.520266 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.520269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.520271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.520335 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.520510 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.520512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.520513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.520697 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:32.520830 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:32.520974 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:42:32.521088 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":139,"threads_started":1}
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[22905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:32.529897 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.523569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.18.235.190:32803)
0927 18:42:32.523688 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:32.523692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:32.523730 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:42:32.523822 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:32.523824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:32.523858 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:32.523915 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:32.524372 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.524841 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.525529 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.525733 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.525799 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.527644 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:32.527647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:32.527649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:32.527669 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:32.528672 (+ 1003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.528994 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.528996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.528997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.529057 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.529265 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.529267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.529268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.529445 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:32.529536 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:32.529672 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:42:32.529762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 18:42:32.530215 22950 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44706
I20260927 18:42:32.530714 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23012
I20260927 18:42:32.535466 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23146
I20260927 18:42:32.539708 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23280
I20260927 18:42:32.543660 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 22918
2026-09-27T18:42:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1914 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[23421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[23421](info): set up 2 sockets
Sep 27 18:42:32 dist-test-slave-6lr6 krb5kdc[23421](info): commencing operation
krb5kdc: starting...
W20260927 18:42:33.484814 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:33 dist-test-slave-6lr6 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:33.589432 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39085
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:33.691649 23434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:33.691861 23434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:33.691903 23434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:33.695873 23434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:33.695925 23434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:33.695945 23434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:33.695963 23434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:33.695987 23434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:33.700904 23434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:33.701771 23434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:33.702670 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:33.710132 23440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.710143 23442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.710203 23439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:33.710191 23434 server_base.cc:1034] running on GCE node
I20260927 18:42:33.710834 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:33.711792 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:33.712971 23434 hybrid_clock.cc:648] HybridClock initialized: now 1790534553712950 us; error 37 us; skew 500 ppm
Sep 27 18:42:33 dist-test-slave-6lr6 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:33.715524 23434 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:33.716144 23434 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:33.716203 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:33.716352 23434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:33.718118 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "eb5886e8a3534b73ace10aeb9c11a0a1"
format_stamp: "Formatted at 2026-09-27 18:42:33 on dist-test-slave-6lr6"
server_key: "c996bd796900ce34df8adf4a25e9d40b"
server_key_iv: "1e0feef6b191b168957a1b9f142858c8"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.718575 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "eb5886e8a3534b73ace10aeb9c11a0a1"
format_stamp: "Formatted at 2026-09-27 18:42:33 on dist-test-slave-6lr6"
server_key: "c996bd796900ce34df8adf4a25e9d40b"
server_key_iv: "1e0feef6b191b168957a1b9f142858c8"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.721709 23434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:33.723824 23448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.724874 23434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:33.724972 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "eb5886e8a3534b73ace10aeb9c11a0a1"
format_stamp: "Formatted at 2026-09-27 18:42:33 on dist-test-slave-6lr6"
server_key: "c996bd796900ce34df8adf4a25e9d40b"
server_key_iv: "1e0feef6b191b168957a1b9f142858c8"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.725055 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:33.753912 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:33.756323 23434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:33.756467 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:33.762564 23434 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:39085
I20260927 18:42:33.762811 23500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:39085 every 8 connection(s)
I20260927 18:42:33.763134 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:33.763836 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23434
I20260927 18:42:33.763936 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:33.764166 19374 external_mini_cluster.cc:1428] Setting key e3bc9753432ae41ef5a0f5600fc3fe21
I20260927 18:42:33.765833 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:33.771579 23501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1: Bootstrap starting.
Sep 27 18:42:33 dist-test-slave-6lr6 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:33.773764 23501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:33.774405 23501 log.cc:824] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:33.776284 23501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1: No bootstrap required, opened a new log
I20260927 18:42:33.777009 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.765398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.18.235.190:39085)
0927 18:42:33.765769 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.765783 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.765815 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:42:33.766432 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.766447 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.766739 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.767001 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:33.767825 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.768761 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.769888 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.770324 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.770431 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.772966 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:33.772970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:33.772982 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:33.773006 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:33.774458 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.774925 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.774931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.774936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.775066 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.775319 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.775321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.775322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.775665 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:33.775790 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.776066 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.776304 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260927 18:42:33.779526 23501 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39085 } }
I20260927 18:42:33.779721 23501 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:33.779773 23501 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5886e8a3534b73ace10aeb9c11a0a1, State: Initialized, Role: FOLLOWER
I20260927 18:42:33.780172 23501 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39085 } }
I20260927 18:42:33.780309 23501 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:33.780371 23501 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:33.780458 23501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:33.781244 23501 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39085 } }
I20260927 18:42:33.781553 23501 leader_election.cc:302] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5886e8a3534b73ace10aeb9c11a0a1; no voters:
I20260927 18:42:33.781819 23501 leader_election.cc:288] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:33.781872 23506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:33.782050 23506 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [term 1 LEADER]: Becoming Leader. State: Replica: eb5886e8a3534b73ace10aeb9c11a0a1, State: Running, Role: LEADER
I20260927 18:42:33.782392 23506 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39085 } }
I20260927 18:42:33.782842 23501 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:33.783776 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5886e8a3534b73ace10aeb9c11a0a1. Latest consensus state: current_term: 1 leader_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39085 } } }
I20260927 18:42:33.783888 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:33.784147 23507 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5886e8a3534b73ace10aeb9c11a0a1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39085 } } }
I20260927 18:42:33.784220 23507 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:33.784255 23515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:33.786980 23515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:33.792201 23515 catalog_manager.cc:1347] Generated new cluster ID: 996b982b61b541c681488985cf4ce332
I20260927 18:42:33.792254 23515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:33.807983 23515 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:33.808766 23515 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:33.815135 23515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1: Generated new TSK 0
I20260927 18:42:33.815655 23515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:33.846251 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39085
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:33.955451 23527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:33.955646 23527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:33.955684 23527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:33.959473 23527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:33.959515 23527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:33.959568 23527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:33.964323 23527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:33.965078 23527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:33.965905 23527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:33.973788 23535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.973884 23532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:33.973980 23527 server_base.cc:1034] running on GCE node
W20260927 18:42:33.973966 23533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:33.974299 23527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:33.974784 23527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:33.975917 23527 hybrid_clock.cc:648] HybridClock initialized: now 1790534553975906 us; error 27 us; skew 500 ppm
Sep 27 18:42:33 dist-test-slave-6lr6 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:33.978312 23527 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:33.978900 23527 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:33.978956 23527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:33.979094 23527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:33.980772 23527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "585cfc0dc5c040f3a64997e782115f9e"
format_stamp: "Formatted at 2026-09-27 18:42:33 on dist-test-slave-6lr6"
server_key: "997aca4faae1820db26a53680af0ddfe"
server_key_iv: "dcbeda7739b585a7d0e102a5e6dcdeab"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.981254 23527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "585cfc0dc5c040f3a64997e782115f9e"
format_stamp: "Formatted at 2026-09-27 18:42:33 on dist-test-slave-6lr6"
server_key: "997aca4faae1820db26a53680af0ddfe"
server_key_iv: "dcbeda7739b585a7d0e102a5e6dcdeab"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.984383 23527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:42:33.986369 23541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.987368 23527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:33.987459 23527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "585cfc0dc5c040f3a64997e782115f9e"
format_stamp: "Formatted at 2026-09-27 18:42:33 on dist-test-slave-6lr6"
server_key: "997aca4faae1820db26a53680af0ddfe"
server_key_iv: "dcbeda7739b585a7d0e102a5e6dcdeab"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.987537 23527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:34.003149 23527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:34.005347 23527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:34.005487 23527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:34.005925 23527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:34.006603 23527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:34.006645 23527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.006690 23527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:34.006718 23527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.014401 23527 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:38105
I20260927 18:42:34.014423 23654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:38105 every 8 connection(s)
I20260927 18:42:34.014859 23527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:34.021193 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23527
I20260927 18:42:34.021294 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:34.021529 19374 external_mini_cluster.cc:1428] Setting key b350e06580cba8279840794220daf7d4
Sep 27 18:42:34 dist-test-slave-6lr6 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:34.027029 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.016282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:35775 (local address 127.18.235.190:39085)
0927 18:42:34.016398 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.016402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.017098 (+ 696us) server_negotiation.cc:407] Connection header received
0927 18:42:34.018264 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.018267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.018318 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.018387 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.019384 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.019925 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.021003 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.021295 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.021389 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.023782 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.023785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.023787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.023808 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.025310 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.025729 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.025732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.025734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.025813 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.026071 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.026073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.026075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.026259 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.026345 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.026747 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.026874 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:42:34.027767 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.016591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39085 (local address 127.18.235.129:35775)
0927 18:42:34.016975 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.017012 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.018081 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.018530 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.018537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.018840 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:34.019251 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.019263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.020037 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.020039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.020875 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.020883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.021519 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.021522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.021684 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.022161 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:34.022177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:34.023635 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:34.025406 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:34.025410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:34.025420 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:34.025641 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:34.025894 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:34.025896 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:34.025897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:34.026004 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:34.026334 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:34.026338 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:34.026524 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.026687 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.026939 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260927 18:42:34.028755 23655 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39085
I20260927 18:42:34.028940 23655 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:34.029340 23655 heartbeater.cc:511] Master 127.18.235.190:39085 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:42:34.030606 23465 ts_manager.cc:194] Registered new tserver with Master: 585cfc0dc5c040f3a64997e782115f9e (127.18.235.129:38105)
I20260927 18:42:34.031744 23465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:35775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:34.046018 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39085
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:34.145599 23660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:34.145808 23660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:34.145865 23660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:34.149699 23660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:34.149745 23660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:34.149821 23660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:34.154635 23660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:34.155483 23660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:34.156347 23660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:42:34.164376 23660 server_base.cc:1034] running on GCE node
W20260927 18:42:34.164294 23666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:34.164314 23668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:34.164295 23665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:34.164923 23660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:34.165486 23660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:34.166632 23660 hybrid_clock.cc:648] HybridClock initialized: now 1790534554166618 us; error 31 us; skew 500 ppm
Sep 27 18:42:34 dist-test-slave-6lr6 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:34.169001 23660 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:34.169613 23660 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:34.169677 23660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:34.169848 23660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:34.171563 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "94ef614c574e4de9847d06c78e5cd6e0"
format_stamp: "Formatted at 2026-09-27 18:42:34 on dist-test-slave-6lr6"
server_key: "94ea527f137bf0f893893102cc4fa56f"
server_key_iv: "424fbbde15668bc3e0a9280794a88740"
server_key_version: "encryptionkey@0"
I20260927 18:42:34.172048 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "94ef614c574e4de9847d06c78e5cd6e0"
format_stamp: "Formatted at 2026-09-27 18:42:34 on dist-test-slave-6lr6"
server_key: "94ea527f137bf0f893893102cc4fa56f"
server_key_iv: "424fbbde15668bc3e0a9280794a88740"
server_key_version: "encryptionkey@0"
I20260927 18:42:34.175204 23660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:34.177284 23674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.178292 23660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:34.178383 23660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "94ef614c574e4de9847d06c78e5cd6e0"
format_stamp: "Formatted at 2026-09-27 18:42:34 on dist-test-slave-6lr6"
server_key: "94ea527f137bf0f893893102cc4fa56f"
server_key_iv: "424fbbde15668bc3e0a9280794a88740"
server_key_version: "encryptionkey@0"
I20260927 18:42:34.178464 23660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:34.208716 23660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:34.211133 23660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:34.211287 23660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:34.211782 23660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:34.212507 23660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:34.212550 23660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.212599 23660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:34.212628 23660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.220340 23660 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:45599
I20260927 18:42:34.220373 23787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:45599 every 8 connection(s)
I20260927 18:42:34.220841 23660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:42:34 dist-test-slave-6lr6 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:34.230526 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23660
I20260927 18:42:34.230624 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:34.230885 19374 external_mini_cluster.cc:1428] Setting key bec078553951dad2b9a31b28e6658f45
I20260927 18:42:34.232784 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.222328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:49839 (local address 127.18.235.190:39085)
0927 18:42:34.222460 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.222464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.223195 (+ 731us) server_negotiation.cc:407] Connection header received
0927 18:42:34.224408 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.224411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.224465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.224538 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.225632 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.226144 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.227198 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.227433 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.227521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.229632 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.229635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.229637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.229661 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.230914 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.231415 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.231417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.231419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.231492 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.231792 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.231795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.231797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.231983 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.232070 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.232575 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.232667 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:42:34.233772 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.222659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39085 (local address 127.18.235.130:49839)
0927 18:42:34.223066 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.223103 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.224221 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.224703 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.224723 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.225037 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:34.225501 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.225513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.226260 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.226263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.227087 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.227096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.227558 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.227562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.227720 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.228184 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:34.228198 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:34.229498 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:34.231014 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:34.231020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:34.231032 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:34.231329 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:34.231577 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:34.231580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:34.231582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:34.231709 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:34.232083 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:34.232089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:34.232348 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.232530 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.232838 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
I20260927 18:42:34.234838 23788 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39085
I20260927 18:42:34.235025 23788 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:34.235459 23788 heartbeater.cc:511] Master 127.18.235.190:39085 requested a full tablet report, sending...
I20260927 18:42:34.236395 23465 ts_manager.cc:194] Registered new tserver with Master: 94ef614c574e4de9847d06c78e5cd6e0 (127.18.235.130:45599)
I20260927 18:42:34.236922 23465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:49839
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:34.255597 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39085
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:34.355950 23793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:34.356163 23793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:34.356217 23793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:34.360019 23793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:34.360064 23793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:34.360141 23793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:34.364974 23793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:34.365820 23793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:34.366652 23793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:34.374687 23799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:34.374714 23798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:34.374715 23801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:34.374804 23793 server_base.cc:1034] running on GCE node
I20260927 18:42:34.375278 23793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:34.375809 23793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:34.376950 23793 hybrid_clock.cc:648] HybridClock initialized: now 1790534554376935 us; error 26 us; skew 500 ppm
Sep 27 18:42:34 dist-test-slave-6lr6 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:34.379311 23793 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:34.379935 23793 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:34.379992 23793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:34.380162 23793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:34.381965 23793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "2b66be5ee7504dd7b49c9a0cf1996336"
format_stamp: "Formatted at 2026-09-27 18:42:34 on dist-test-slave-6lr6"
server_key: "dcd367991f3c0141f4fa0eac430645a5"
server_key_iv: "94aebdcb0c0a41c32803941e1e46285e"
server_key_version: "encryptionkey@0"
I20260927 18:42:34.382443 23793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "2b66be5ee7504dd7b49c9a0cf1996336"
format_stamp: "Formatted at 2026-09-27 18:42:34 on dist-test-slave-6lr6"
server_key: "dcd367991f3c0141f4fa0eac430645a5"
server_key_iv: "94aebdcb0c0a41c32803941e1e46285e"
server_key_version: "encryptionkey@0"
I20260927 18:42:34.385751 23793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:34.387790 23807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.389140 23793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:34.389241 23793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "2b66be5ee7504dd7b49c9a0cf1996336"
format_stamp: "Formatted at 2026-09-27 18:42:34 on dist-test-slave-6lr6"
server_key: "dcd367991f3c0141f4fa0eac430645a5"
server_key_iv: "94aebdcb0c0a41c32803941e1e46285e"
server_key_version: "encryptionkey@0"
I20260927 18:42:34.389341 23793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:34.408674 23793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:34.411012 23793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:34.411139 23793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:34.411567 23793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:34.412246 23793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:34.412288 23793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.412321 23793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:34.412341 23793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:34.419936 23793 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:42185
I20260927 18:42:34.419984 23920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:42185 every 8 connection(s)
I20260927 18:42:34.420423 23793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:42:34 dist-test-slave-6lr6 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:34.429984 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23793
I20260927 18:42:34.430075 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:34.430353 19374 external_mini_cluster.cc:1428] Setting key f6f94db335162b6bded02486692c6f8f
I20260927 18:42:34.432042 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.421894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:45739 (local address 127.18.235.190:39085)
0927 18:42:34.422015 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.422018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.422736 (+ 718us) server_negotiation.cc:407] Connection header received
0927 18:42:34.423890 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.423894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.423947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.424020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.425062 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.425606 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.426649 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.426870 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.426957 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.429030 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.429033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.429035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.429059 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.430158 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.430674 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.430677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.430679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.430788 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.431123 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.431126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.431127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.431308 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.431399 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.431836 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.431934 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:34.432857 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.422214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39085 (local address 127.18.235.131:45739)
0927 18:42:34.422601 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.422637 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.423705 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.424186 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.424193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.424510 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:34.424927 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.424940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.425717 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.425721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.426538 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.426547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.426996 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.427000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.427143 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.427598 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:34.427613 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:34.428897 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:34.430258 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:34.430263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:34.430275 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:34.430584 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:34.430877 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:34.430880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:34.430882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:34.431042 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:34.431404 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:34.431408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:34.431622 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.431807 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.432095 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20260927 18:42:34.433733 23921 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39085
I20260927 18:42:34.433923 23921 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:34.434355 23921 heartbeater.cc:511] Master 127.18.235.190:39085 requested a full tablet report, sending...
I20260927 18:42:34.435189 23465 ts_manager.cc:194] Registered new tserver with Master: 2b66be5ee7504dd7b49c9a0cf1996336 (127.18.235.131:42185)
I20260927 18:42:34.435719 23465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:45739
I20260927 18:42:34.443696 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:34.451159 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.445347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.18.235.190:39085)
0927 18:42:34.445462 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.445466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.445590 (+ 124us) server_negotiation.cc:407] Connection header received
0927 18:42:34.445680 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.445682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.445725 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.445778 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.446303 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.446765 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.447456 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.447644 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.447704 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.448873 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.448876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.448878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.448896 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.449973 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.450294 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.450296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.450297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.450354 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.450522 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.450524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.450525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.450691 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.450793 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.450947 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.451025 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:34.453419 23465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:34.456107 23465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:34.468560 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.463234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:45599 (local address 127.0.0.1:38666)
0927 18:42:34.463673 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.463734 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.463848 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.464545 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.464550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.464572 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:34.464679 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.464686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.466047 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.466051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.467251 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.467260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.468088 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.468091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.468193 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.468228 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.468332 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.468437 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":64,"threads_started":1}
I20260927 18:42:34.468603 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.463395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.18.235.130:45599)
0927 18:42:34.463792 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.463803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.463825 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:34.464034 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.464040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.464228 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.464372 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:34.464794 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.465913 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.467372 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.467975 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.468089 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.468202 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.468368 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.468464 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":93,"threads_started":1}
I20260927 18:42:34.469008 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.463370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:38105 (local address 127.0.0.1:54224)
0927 18:42:34.464364 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.464377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.464460 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.465277 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.465280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.465296 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:34.465385 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.465391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.466479 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.466482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.467661 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.467671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.468657 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.468661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.468759 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.468781 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.468831 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.468894 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":78,"threads_started":1}
I20260927 18:42:34.469376 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.464012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:42185 (local address 127.0.0.1:56926)
0927 18:42:34.464854 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.464863 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.464932 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.465443 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.465446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.465460 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:34.465529 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.465533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.466619 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.466622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.467715 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.467723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.468960 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.468963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.469024 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.469040 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.469209 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.469276 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":69,"threads_started":1}
I20260927 18:42:34.469610 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.463989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.18.235.129:38105)
0927 18:42:34.464854 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.464864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.464885 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:42:34.464951 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.464955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.465155 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.465291 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:34.465490 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.466366 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.467783 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.468523 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.469214 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.469335 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.469381 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.469469 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":99,"threads_started":1}
I20260927 18:42:34.469936 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.464315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.18.235.131:42185)
0927 18:42:34.464675 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.464685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.464943 (+ 258us) server_negotiation.cc:407] Connection header received
0927 18:42:34.465012 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.465016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.465182 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.465315 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:34.465622 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.466499 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.467828 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.468497 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.469568 (+ 1071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.469684 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.469728 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.469801 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260927 18:42:34.471439 23722 tablet_service.cc:1467] Processing CreateTablet for tablet 667a3e6e18d64da3b8d5f987db259de6 (DEFAULT_TABLE table=test-table [id=2dd11428931f40e1a9c76f335432d6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:34.472309 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667a3e6e18d64da3b8d5f987db259de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:34.472254 23855 tablet_service.cc:1467] Processing CreateTablet for tablet 667a3e6e18d64da3b8d5f987db259de6 (DEFAULT_TABLE table=test-table [id=2dd11428931f40e1a9c76f335432d6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:34.473256 23855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667a3e6e18d64da3b8d5f987db259de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:34.472973 23589 tablet_service.cc:1467] Processing CreateTablet for tablet 667a3e6e18d64da3b8d5f987db259de6 (DEFAULT_TABLE table=test-table [id=2dd11428931f40e1a9c76f335432d6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:34.473922 23589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667a3e6e18d64da3b8d5f987db259de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:34.477826 23938 tablet_bootstrap.cc:492] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Bootstrap starting.
I20260927 18:42:34.479646 23939 tablet_bootstrap.cc:492] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Bootstrap starting.
I20260927 18:42:34.480381 23938 tablet_bootstrap.cc:654] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:34.480939 23940 tablet_bootstrap.cc:492] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: Bootstrap starting.
I20260927 18:42:34.481599 23938 log.cc:824] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:34.481700 23939 tablet_bootstrap.cc:654] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:34.482674 23940 tablet_bootstrap.cc:654] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:34.482779 23939 log.cc:824] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:34.483567 23938 tablet_bootstrap.cc:492] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: No bootstrap required, opened a new log
I20260927 18:42:34.483758 23938 ts_tablet_manager.cc:1397] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:42:34.483809 23940 log.cc:824] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:34.484690 23939 tablet_bootstrap.cc:492] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: No bootstrap required, opened a new log
I20260927 18:42:34.484871 23939 ts_tablet_manager.cc:1397] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:34.485602 23940 tablet_bootstrap.cc:492] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: No bootstrap required, opened a new log
I20260927 18:42:34.485764 23940 ts_tablet_manager.cc:1397] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:34.486814 23938 raft_consensus.cc:348] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.487012 23938 raft_consensus.cc:374] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:34.487079 23938 raft_consensus.cc:729] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef614c574e4de9847d06c78e5cd6e0, State: Initialized, Role: FOLLOWER
I20260927 18:42:34.487450 23938 consensus_queue.cc:260] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.488001 23788 heartbeater.cc:503] Master 127.18.235.190:39085 was elected leader, sending a full tablet report...
I20260927 18:42:34.488336 23938 ts_tablet_manager.cc:1428] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:34.488384 23939 raft_consensus.cc:348] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.488610 23939 raft_consensus.cc:374] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:34.488649 23939 raft_consensus.cc:729] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b66be5ee7504dd7b49c9a0cf1996336, State: Initialized, Role: FOLLOWER
I20260927 18:42:34.489073 23939 consensus_queue.cc:260] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.489184 23940 raft_consensus.cc:348] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.489471 23940 raft_consensus.cc:374] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:34.489552 23940 raft_consensus.cc:729] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585cfc0dc5c040f3a64997e782115f9e, State: Initialized, Role: FOLLOWER
I20260927 18:42:34.489657 23921 heartbeater.cc:503] Master 127.18.235.190:39085 was elected leader, sending a full tablet report...
I20260927 18:42:34.489961 23940 consensus_queue.cc:260] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.490051 23939 ts_tablet_manager.cc:1428] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:34.490463 23655 heartbeater.cc:503] Master 127.18.235.190:39085 was elected leader, sending a full tablet report...
I20260927 18:42:34.490823 23940 ts_tablet_manager.cc:1428] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 18:42:34.515666 23656 tablet.cc:2369] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:34.671141 23922 tablet.cc:2369] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:34.721776 23789 tablet.cc:2369] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:34.780045 23946 raft_consensus.cc:482] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:34.780205 23946 raft_consensus.cc:504] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.781247 23946 leader_election.cc:288] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ef614c574e4de9847d06c78e5cd6e0 (127.18.235.130:45599), 2b66be5ee7504dd7b49c9a0cf1996336 (127.18.235.131:42185)
I20260927 18:42:34.786018 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.781584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:44561 (local address 127.18.235.130:45599)
0927 18:42:34.781700 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.781704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.781785 (+ 81us) server_negotiation.cc:407] Connection header received
0927 18:42:34.781930 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.781933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.781978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.782051 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:34.782300 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.783099 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.784829 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.785453 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.785561 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.785610 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.785838 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.785902 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:34.786018 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.781588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:42269 (local address 127.18.235.131:42185)
0927 18:42:34.781719 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.781723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.781917 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:42:34.782029 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.782033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.782076 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.782142 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:34.782393 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.783132 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.784828 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.785447 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.785561 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.785610 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.785838 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.785902 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:42:34.786065 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.781410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:45599 (local address 127.18.235.129:44561)
0927 18:42:34.781698 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.781722 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.781820 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.782056 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.782058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.782068 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:34.782204 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.782211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.783195 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.783198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.784703 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.784714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.785552 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.785556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.785698 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.785736 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.785804 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.785907 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":122,"thread_start_us":74,"threads_started":1}
I20260927 18:42:34.786067 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.781480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:42185 (local address 127.18.235.129:42269)
0927 18:42:34.781840 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.781851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:34.781942 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:34.782161 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:34.782164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:34.782177 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:34.782306 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.782312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.783209 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.783212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.784709 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:34.784715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.785552 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.785556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:34.785705 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.785737 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:42:34.785804 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:42:34.785906 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":162,"thread_start_us":105,"threads_started":1}
I20260927 18:42:34.786631 23742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a3e6e18d64da3b8d5f987db259de6" candidate_uuid: "585cfc0dc5c040f3a64997e782115f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef614c574e4de9847d06c78e5cd6e0" is_pre_election: true
I20260927 18:42:34.786684 23875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a3e6e18d64da3b8d5f987db259de6" candidate_uuid: "585cfc0dc5c040f3a64997e782115f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" is_pre_election: true
I20260927 18:42:34.786877 23742 raft_consensus.cc:2454] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585cfc0dc5c040f3a64997e782115f9e in term 0.
I20260927 18:42:34.786947 23875 raft_consensus.cc:2454] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585cfc0dc5c040f3a64997e782115f9e in term 0.
I20260927 18:42:34.787276 23545 leader_election.cc:302] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585cfc0dc5c040f3a64997e782115f9e, 94ef614c574e4de9847d06c78e5cd6e0; no voters:
I20260927 18:42:34.787508 23946 raft_consensus.cc:2781] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:34.787583 23946 raft_consensus.cc:482] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:34.787629 23946 raft_consensus.cc:3037] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:34.788540 23946 raft_consensus.cc:504] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.788805 23946 leader_election.cc:288] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [CANDIDATE]: Term 1 election: Requested vote from peers 94ef614c574e4de9847d06c78e5cd6e0 (127.18.235.130:45599), 2b66be5ee7504dd7b49c9a0cf1996336 (127.18.235.131:42185)
I20260927 18:42:34.789072 23742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a3e6e18d64da3b8d5f987db259de6" candidate_uuid: "585cfc0dc5c040f3a64997e782115f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef614c574e4de9847d06c78e5cd6e0"
I20260927 18:42:34.789126 23875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a3e6e18d64da3b8d5f987db259de6" candidate_uuid: "585cfc0dc5c040f3a64997e782115f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b66be5ee7504dd7b49c9a0cf1996336"
I20260927 18:42:34.789223 23742 raft_consensus.cc:3037] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:34.789263 23875 raft_consensus.cc:3037] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:34.790385 23742 raft_consensus.cc:2454] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585cfc0dc5c040f3a64997e782115f9e in term 1.
I20260927 18:42:34.790385 23875 raft_consensus.cc:2454] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585cfc0dc5c040f3a64997e782115f9e in term 1.
I20260927 18:42:34.790639 23545 leader_election.cc:302] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b66be5ee7504dd7b49c9a0cf1996336, 585cfc0dc5c040f3a64997e782115f9e, 94ef614c574e4de9847d06c78e5cd6e0; no voters:
I20260927 18:42:34.790791 23946 raft_consensus.cc:2781] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:34.790970 23946 raft_consensus.cc:686] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 1 LEADER]: Becoming Leader. State: Replica: 585cfc0dc5c040f3a64997e782115f9e, State: Running, Role: LEADER
I20260927 18:42:34.791270 23946 consensus_queue.cc:237] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } }
I20260927 18:42:34.793948 23462 catalog_manager.cc:5495] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 585cfc0dc5c040f3a64997e782115f9e (127.18.235.129). New cstate: current_term: 1 leader_uuid: "585cfc0dc5c040f3a64997e782115f9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585cfc0dc5c040f3a64997e782115f9e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:34.824518 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.821401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.18.235.129:38105)
0927 18:42:34.821505 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.821509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.821521 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:34.821671 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.821674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.821719 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.821798 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:34.822032 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.822502 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.823288 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.823491 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.823570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.823831 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:34.824186 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:34.824284 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.824356 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.824406 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:42:34.829502 23742 raft_consensus.cc:1261] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 1 FOLLOWER]: Refusing update from remote peer 585cfc0dc5c040f3a64997e782115f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:34.829502 23875 raft_consensus.cc:1261] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 1 FOLLOWER]: Refusing update from remote peer 585cfc0dc5c040f3a64997e782115f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:34.830082 23946 consensus_queue.cc:1035] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b66be5ee7504dd7b49c9a0cf1996336" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 42185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:34.830283 23949 consensus_queue.cc:1035] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef614c574e4de9847d06c78e5cd6e0" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:34.836987 23954 mvcc.cc:205] Tried to move back new op lower bound from 7334029536575660032 to 7334029536427249664. Current Snapshot: MvccSnapshot[applied={T|T < 7334029536575660032}]
I20260927 18:42:34.838604 23953 mvcc.cc:205] Tried to move back new op lower bound from 7334029536575660032 to 7334029536427249664. Current Snapshot: MvccSnapshot[applied={T|T < 7334029536575660032}]
I20260927 18:42:34.843782 23464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:34.843905 23464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:34.845849 23464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb5886e8a3534b73ace10aeb9c11a0a1: Sending DeleteTablet for 3 replicas of tablet 667a3e6e18d64da3b8d5f987db259de6
I20260927 18:42:34.846487 23589 tablet_service.cc:1514] Processing DeleteTablet for tablet 667a3e6e18d64da3b8d5f987db259de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:34 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:54224
I20260927 18:42:34.846487 23722 tablet_service.cc:1514] Processing DeleteTablet for tablet 667a3e6e18d64da3b8d5f987db259de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:34 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:38666
I20260927 18:42:34.846611 23855 tablet_service.cc:1514] Processing DeleteTablet for tablet 667a3e6e18d64da3b8d5f987db259de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:34 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:56926
I20260927 18:42:34.846876 23961 tablet_replica.cc:330] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e: stopping tablet replica
I20260927 18:42:34.847137 23961 raft_consensus.cc:2229] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:34.847297 23962 tablet_replica.cc:330] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: stopping tablet replica
I20260927 18:42:34.847419 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23527
I20260927 18:42:34.847440 23961 raft_consensus.cc:2258] T 667a3e6e18d64da3b8d5f987db259de6 P 585cfc0dc5c040f3a64997e782115f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:34.848372 23963 tablet_replica.cc:330] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: stopping tablet replica
I20260927 18:42:34.848611 23963 raft_consensus.cc:2229] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:34.848814 23962 raft_consensus.cc:2229] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:34.848862 23963 raft_consensus.cc:2258] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:34.849015 23962 raft_consensus.cc:2258] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:34.850093 23962 ts_tablet_manager.cc:1905] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:34.852041 23962 ts_tablet_manager.cc:1918] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:34.852113 23962 log.cc:1196] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/667a3e6e18d64da3b8d5f987db259de6
I20260927 18:42:34.852348 23962 ts_tablet_manager.cc:1939] T 667a3e6e18d64da3b8d5f987db259de6 P 94ef614c574e4de9847d06c78e5cd6e0: Deleting consensus metadata
W20260927 18:42:34.852788 23449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:38105 (error 108)
I20260927 18:42:34.853618 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.853390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:38105 (local address 127.0.0.1:48912)
0927 18:42:34.853490 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.853548 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:38105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:42:34.853693 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23660
I20260927 18:42:34.853669 23452 catalog_manager.cc:4841] TS 94ef614c574e4de9847d06c78e5cd6e0 (127.18.235.130:45599): tablet 667a3e6e18d64da3b8d5f987db259de6 (table test-table [id=2dd11428931f40e1a9c76f335432d6b4]) successfully deleted
I20260927 18:42:34.854106 23963 ts_tablet_manager.cc:1905] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:42:34.855796 23449 catalog_manager.cc:4584] TS 585cfc0dc5c040f3a64997e782115f9e (127.18.235.129:38105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 667a3e6e18d64da3b8d5f987db259de6: Network error: Client connection negotiation failed: client connection to 127.18.235.129:38105: connect: Connection refused (error 111)
I20260927 18:42:34.856012 23963 ts_tablet_manager.cc:1918] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:34.856087 23963 log.cc:1196] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/667a3e6e18d64da3b8d5f987db259de6
I20260927 18:42:34.856323 23963 ts_tablet_manager.cc:1939] T 667a3e6e18d64da3b8d5f987db259de6 P 2b66be5ee7504dd7b49c9a0cf1996336: Deleting consensus metadata
I20260927 18:42:34.857179 23451 catalog_manager.cc:4841] TS 2b66be5ee7504dd7b49c9a0cf1996336 (127.18.235.131:42185): tablet 667a3e6e18d64da3b8d5f987db259de6 (table test-table [id=2dd11428931f40e1a9c76f335432d6b4]) successfully deleted
I20260927 18:42:34.859442 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23793
I20260927 18:42:34.863510 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23434
2026-09-27T18:42:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2319 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T18:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:34Z Disabled control of system clock
I20260927 18:42:34.889134 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39895
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:42:34.989671 23968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:34.989886 23968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:34.989918 23968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:34.993732 23968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:34.993774 23968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:34.993793 23968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:34.993814 23968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:34.993829 23968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:34.998245 23968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:34.999032 23968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:35.000159 23968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:35.007452 23973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:35.007452 23974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.007520 23968 server_base.cc:1034] running on GCE node
W20260927 18:42:35.007478 23976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.008150 23968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:35.009095 23968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:35.010286 23968 hybrid_clock.cc:648] HybridClock initialized: now 1790534555010270 us; error 27 us; skew 500 ppm
I20260927 18:42:35.011885 23968 webserver.cc:460] Webserver started at http://127.18.235.190:33533/ using document root <none> and password file <none>
I20260927 18:42:35.012377 23968 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:35.012436 23968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:35.012617 23968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:35.014485 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "170caf16f42348c1bff8d90750faa4b1"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "10766c3a826009c4afe5e87399df139e"
server_key_iv: "511642e0ad1ce35254d5ac3060332533"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.014981 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "170caf16f42348c1bff8d90750faa4b1"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "10766c3a826009c4afe5e87399df139e"
server_key_iv: "511642e0ad1ce35254d5ac3060332533"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.018208 23968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:35.020221 23982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.021329 23968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:35.021428 23968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "170caf16f42348c1bff8d90750faa4b1"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "10766c3a826009c4afe5e87399df139e"
server_key_iv: "511642e0ad1ce35254d5ac3060332533"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.021512 23968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:35.025548 23968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:35.026063 23968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:35.026209 23968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:35.031983 23968 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:39895
I20260927 18:42:35.031989 24034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:39895 every 8 connection(s)
I20260927 18:42:35.032750 23968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:35.033139 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 23968
I20260927 18:42:35.033246 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:35.033533 19374 external_mini_cluster.cc:1428] Setting key 3a5c4610a84a23ee85cfc259b3f539b4
I20260927 18:42:35.035763 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:35.038937 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.034744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.18.235.190:39895)
0927 18:42:35.035810 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.035827 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.035866 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:42:35.036518 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.036535 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.036846 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.037144 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.037856 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.037860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.037871 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.037881 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.037929 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.037943 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.038058 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.038130 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.038351 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":628,"threads_started":1}
I20260927 18:42:35.041237 24035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1: Bootstrap starting.
I20260927 18:42:35.043292 24035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:35.043931 24035 log.cc:824] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:35.045516 24035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1: No bootstrap required, opened a new log
I20260927 18:42:35.048547 24035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170caf16f42348c1bff8d90750faa4b1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39895 } }
I20260927 18:42:35.048725 24035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:35.048775 24035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170caf16f42348c1bff8d90750faa4b1, State: Initialized, Role: FOLLOWER
I20260927 18:42:35.049201 24035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170caf16f42348c1bff8d90750faa4b1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39895 } }
I20260927 18:42:35.049350 24035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:35.049419 24035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:35.049517 24035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:35.050247 24035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170caf16f42348c1bff8d90750faa4b1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39895 } }
I20260927 18:42:35.050504 24035 leader_election.cc:302] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 170caf16f42348c1bff8d90750faa4b1; no voters:
I20260927 18:42:35.050719 24035 leader_election.cc:288] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:35.050812 24040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:35.050994 24040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [term 1 LEADER]: Becoming Leader. State: Replica: 170caf16f42348c1bff8d90750faa4b1, State: Running, Role: LEADER
I20260927 18:42:35.051275 24040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170caf16f42348c1bff8d90750faa4b1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39895 } }
I20260927 18:42:35.051690 24035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:35.052683 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170caf16f42348c1bff8d90750faa4b1. Latest consensus state: current_term: 1 leader_uuid: "170caf16f42348c1bff8d90750faa4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170caf16f42348c1bff8d90750faa4b1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39895 } } }
I20260927 18:42:35.052803 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:35.053043 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "170caf16f42348c1bff8d90750faa4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170caf16f42348c1bff8d90750faa4b1" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39895 } } }
I20260927 18:42:35.053150 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:35.053566 24049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:35.055946 24049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:35.060187 24049 catalog_manager.cc:1347] Generated new cluster ID: dd7d2e49841b4afba1918f8d6b3b64e4
I20260927 18:42:35.060242 24049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:35.070608 24049 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:35.070847 24049 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:35.081003 24049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1: Generated new TSK 0
I20260927 18:42:35.081534 24049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:42:35.089345 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39895
--builtin_ntp_servers=127.18.235.148:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:42:35.190860 24059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:35.191071 24059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:35.191107 24059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:35.195039 24059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:35.195080 24059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:35.195137 24059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:35.199647 24059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:35.200526 24059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:35.201812 24059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:35.210450 24064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:35.210485 24067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.210587 24059 server_base.cc:1034] running on GCE node
W20260927 18:42:35.210456 24065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.211037 24059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:35.211604 24059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:35.212749 24059 hybrid_clock.cc:648] HybridClock initialized: now 1790534555212727 us; error 39 us; skew 500 ppm
I20260927 18:42:35.214381 24059 webserver.cc:460] Webserver started at http://127.18.235.129:45155/ using document root <none> and password file <none>
I20260927 18:42:35.214901 24059 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:35.214967 24059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:35.215137 24059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:35.216965 24059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "54e8fbacf5f74e329afba21f842adf13"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "c123d42653408fc5cba9380b2d9eeae7"
server_key_iv: "ea842ff4af9c536335eee6d4bf669aa7"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.217499 24059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "54e8fbacf5f74e329afba21f842adf13"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "c123d42653408fc5cba9380b2d9eeae7"
server_key_iv: "ea842ff4af9c536335eee6d4bf669aa7"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.220651 24059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:42:35.222834 24073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.223917 24059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:35.224020 24059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "54e8fbacf5f74e329afba21f842adf13"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "c123d42653408fc5cba9380b2d9eeae7"
server_key_iv: "ea842ff4af9c536335eee6d4bf669aa7"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.224104 24059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:35.228165 24059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:35.228644 24059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:35.228791 24059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:35.229272 24059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:35.230041 24059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:35.230083 24059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.230120 24059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:35.230149 24059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.237757 24059 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:40507
I20260927 18:42:35.237783 24186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:40507 every 8 connection(s)
I20260927 18:42:35.238481 24059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:35.243445 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.239955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:43187 (local address 127.18.235.190:39895)
0927 18:42:35.240071 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.240075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.240950 (+ 875us) server_negotiation.cc:407] Connection header received
0927 18:42:35.242139 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.242142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.242177 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.242244 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.243123 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.243126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.243128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.243138 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.243157 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.243165 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.243231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.243317 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.243349 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:42:35.244150 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.240248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39895 (local address 127.18.235.129:43187)
0927 18:42:35.240827 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.240862 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.241975 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.242377 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.242383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.242463 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.242903 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.242904 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.242937 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.242998 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.243225 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.243229 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.243233 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.243344 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.243509 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260927 18:42:35.244185 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24059
I20260927 18:42:35.244288 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:35.244522 19374 external_mini_cluster.cc:1428] Setting key eb09fe0c796aa5efe183122107b4c0cd
I20260927 18:42:35.245309 24187 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39895
I20260927 18:42:35.245503 24187 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:35.245886 24187 heartbeater.cc:511] Master 127.18.235.190:39895 requested a full tablet report, sending...
I20260927 18:42:35.246249 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39895
--builtin_ntp_servers=127.18.235.148:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:42:35.247231 23999 ts_manager.cc:194] Registered new tserver with Master: 54e8fbacf5f74e329afba21f842adf13 (127.18.235.129:40507)
W20260927 18:42:35.346920 24190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:35.347119 24190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:35.347154 24190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:35.350916 24190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:35.350953 24190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:35.351002 24190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:35.355527 24190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:35.356314 24190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:35.357463 24190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:35.365633 24195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.365721 24190 server_base.cc:1034] running on GCE node
W20260927 18:42:35.365638 24196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:35.365638 24198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.366192 24190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:35.366709 24190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:35.367861 24190 hybrid_clock.cc:648] HybridClock initialized: now 1790534555367850 us; error 30 us; skew 500 ppm
I20260927 18:42:35.369469 24190 webserver.cc:460] Webserver started at http://127.18.235.130:33313/ using document root <none> and password file <none>
I20260927 18:42:35.369947 24190 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:35.370009 24190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:35.370169 24190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:35.371940 24190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "15bc58a498904f08874532300d0e6e68"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "9b61040f088b5f4167af886752989bcd"
server_key_iv: "5a62abf3ce6065825a498be55b0019dd"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.372426 24190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "15bc58a498904f08874532300d0e6e68"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "9b61040f088b5f4167af886752989bcd"
server_key_iv: "5a62abf3ce6065825a498be55b0019dd"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.375713 24190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:42:35.377835 24204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.378861 24190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:35.378957 24190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "15bc58a498904f08874532300d0e6e68"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "9b61040f088b5f4167af886752989bcd"
server_key_iv: "5a62abf3ce6065825a498be55b0019dd"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.379040 24190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:35.383088 24190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:35.383595 24190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:35.383734 24190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:35.384178 24190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:35.384950 24190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:35.384995 24190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.385041 24190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:35.385068 24190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.392822 24190 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:32973
I20260927 18:42:35.392844 24317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:32973 every 8 connection(s)
I20260927 18:42:35.393604 24190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:35.398574 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.395116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:44049 (local address 127.18.235.190:39895)
0927 18:42:35.395243 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.395247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.396152 (+ 905us) server_negotiation.cc:407] Connection header received
0927 18:42:35.397291 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.397294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.397327 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.397395 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.398246 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.398249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.398251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.398263 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.398282 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.398290 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.398357 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.398445 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.398478 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:42:35.399287 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.395459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39895 (local address 127.18.235.130:44049)
0927 18:42:35.396027 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.396061 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.397127 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.397520 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.397526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.397603 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.398028 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.398030 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.398062 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.398122 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.398348 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.398352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.398355 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.398463 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.398625 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260927 18:42:35.400243 24318 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39895
I20260927 18:42:35.400420 24318 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:35.400745 24318 heartbeater.cc:511] Master 127.18.235.190:39895 requested a full tablet report, sending...
I20260927 18:42:35.400978 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24190
I20260927 18:42:35.401069 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:35.401379 19374 external_mini_cluster.cc:1428] Setting key b14b2e2522a1756b4d85a24d78b2b1e7
I20260927 18:42:35.401721 23999 ts_manager.cc:194] Registered new tserver with Master: 15bc58a498904f08874532300d0e6e68 (127.18.235.130:32973)
I20260927 18:42:35.403160 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39895
--builtin_ntp_servers=127.18.235.148:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:42:35.504158 24321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:35.504366 24321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:35.504403 24321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:35.508313 24321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:35.508354 24321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:35.508416 24321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:35.512871 24321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:35.513713 24321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:35.514890 24321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:42:35.523257 24321 server_base.cc:1034] running on GCE node
W20260927 18:42:35.523180 24326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:35.523178 24329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:35.523167 24327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:35.523778 24321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:35.524291 24321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:35.525434 24321 hybrid_clock.cc:648] HybridClock initialized: now 1790534555525401 us; error 45 us; skew 500 ppm
I20260927 18:42:35.527032 24321 webserver.cc:460] Webserver started at http://127.18.235.131:32895/ using document root <none> and password file <none>
I20260927 18:42:35.527508 24321 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:35.527571 24321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:35.527766 24321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:35.529698 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "770df888cdd24c6ea908420fdc8938df"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "de4a8437ea14819bcdbf0e28f07c401c"
server_key_iv: "57a5b236cdeca290ed8cb62e9067acda"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.530149 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "770df888cdd24c6ea908420fdc8938df"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "de4a8437ea14819bcdbf0e28f07c401c"
server_key_iv: "57a5b236cdeca290ed8cb62e9067acda"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.533228 24321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:35.535318 24335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.536355 24321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:35.536435 24321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "770df888cdd24c6ea908420fdc8938df"
format_stamp: "Formatted at 2026-09-27 18:42:35 on dist-test-slave-6lr6"
server_key: "de4a8437ea14819bcdbf0e28f07c401c"
server_key_iv: "57a5b236cdeca290ed8cb62e9067acda"
server_key_version: "encryptionkey@0"
I20260927 18:42:35.536507 24321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:35.540417 24321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:35.540880 24321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:35.540994 24321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:35.541455 24321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:35.542179 24321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:35.542225 24321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.542265 24321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:35.542284 24321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:35.549691 24321 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:44263
I20260927 18:42:35.549746 24448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:44263 every 8 connection(s)
I20260927 18:42:35.550405 24321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:35.558030 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24321
I20260927 18:42:35.558125 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:35.558351 19374 external_mini_cluster.cc:1428] Setting key f460ae1dc03eabb1e7952402da566a36
I20260927 18:42:35.558374 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.551863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:58487 (local address 127.18.235.190:39895)
0927 18:42:35.551972 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.551976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.552939 (+ 963us) server_negotiation.cc:407] Connection header received
0927 18:42:35.554167 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.554170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.554214 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.554283 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.557959 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.557962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.557964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.557976 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.557994 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.558002 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.558076 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.558205 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.558248 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:42:35.559209 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.552167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39895 (local address 127.18.235.131:58487)
0927 18:42:35.552812 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.552848 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.553982 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.554433 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.554439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.554519 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.557703 (+ 3184us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.557704 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.557738 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.557802 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.558068 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.558076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.558081 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.558233 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.558453 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260927 18:42:35.560156 24449 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39895
I20260927 18:42:35.560359 24449 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:35.560686 24449 heartbeater.cc:511] Master 127.18.235.190:39895 requested a full tablet report, sending...
I20260927 18:42:35.561546 23998 ts_manager.cc:194] Registered new tserver with Master: 770df888cdd24c6ea908420fdc8938df (127.18.235.131:44263)
I20260927 18:42:35.571537 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:35.574915 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.573361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.18.235.190:39895)
0927 18:42:35.573466 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.573469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.573672 (+ 203us) server_negotiation.cc:407] Connection header received
0927 18:42:35.573800 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.573803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.573839 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.573909 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.574582 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.574586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.574589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.574600 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.574620 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.574629 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.574707 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.574771 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.574805 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:42:35.577248 23998 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:35.579878 23998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:35.589668 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.586938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:32973 (local address 127.0.0.1:55924)
0927 18:42:35.587453 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.587495 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.587876 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.588320 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.588325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.588391 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.588851 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.588854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.588873 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.588940 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.589362 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.589370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.589376 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.589477 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.589538 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":49,"threads_started":1}
I20260927 18:42:35.589684 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.587561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.18.235.131:44263)
0927 18:42:35.588282 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.588292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.588310 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:42:35.588350 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.588355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.588500 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.588658 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.589204 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.589209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.589219 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.589228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.589263 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.589286 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.589378 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.589495 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.589559 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":112,"threads_started":1}
I20260927 18:42:35.589674 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.587065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44263 (local address 127.0.0.1:59408)
0927 18:42:35.587467 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.587494 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.587578 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.588639 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.588642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.588654 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.588957 (+ 303us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.588959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.588969 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.588995 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.589362 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.589370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.589376 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.589476 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.589548 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260927 18:42:35.590224 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.587029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.18.235.130:32973)
0927 18:42:35.587829 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.587852 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.587877 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:42:35.587994 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.588000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.588185 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.588311 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.589083 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.589089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.589125 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.589137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.589182 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.589228 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.589324 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.589850 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.590084 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":101,"threads_started":1}
I20260927 18:42:35.590420 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.587276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40507 (local address 127.0.0.1:59028)
0927 18:42:35.588010 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.588022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.588078 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.588490 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.588493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.588505 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.589301 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.589303 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.589312 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.589335 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.590102 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.590106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.590110 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.590282 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.590317 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":73,"threads_started":1}
I20260927 18:42:35.591193 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.587603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.18.235.129:40507)
0927 18:42:35.588030 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.588042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.588067 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:42:35.588197 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.588202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.588354 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.589217 (+ 863us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.589924 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.589929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.589939 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.589948 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.589984 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.590006 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.590998 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.591039 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.591086 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260927 18:42:35.592571 24252 tablet_service.cc:1467] Processing CreateTablet for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd (DEFAULT_TABLE table=test-table [id=7a48b6d6c7394b14b3b91500fe14401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:35.592571 24383 tablet_service.cc:1467] Processing CreateTablet for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd (DEFAULT_TABLE table=test-table [id=7a48b6d6c7394b14b3b91500fe14401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:35.593627 24252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:35.593627 24383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:35.593643 24121 tablet_service.cc:1467] Processing CreateTablet for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd (DEFAULT_TABLE table=test-table [id=7a48b6d6c7394b14b3b91500fe14401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:35.594560 24121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:35.600011 24466 tablet_bootstrap.cc:492] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: Bootstrap starting.
I20260927 18:42:35.601433 24468 tablet_bootstrap.cc:492] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Bootstrap starting.
I20260927 18:42:35.602344 24466 tablet_bootstrap.cc:654] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:35.603236 24466 log.cc:824] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:35.603521 24468 tablet_bootstrap.cc:654] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:35.603935 24467 tablet_bootstrap.cc:492] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Bootstrap starting.
I20260927 18:42:35.604360 24468 log.cc:824] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:35.604941 24466 tablet_bootstrap.cc:492] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: No bootstrap required, opened a new log
I20260927 18:42:35.605126 24466 ts_tablet_manager.cc:1397] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:35.605767 24467 tablet_bootstrap.cc:654] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:35.606385 24467 log.cc:824] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:35.607987 24468 tablet_bootstrap.cc:492] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: No bootstrap required, opened a new log
I20260927 18:42:35.608162 24468 ts_tablet_manager.cc:1397] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 18:42:35.608196 24467 tablet_bootstrap.cc:492] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: No bootstrap required, opened a new log
I20260927 18:42:35.608348 24467 ts_tablet_manager.cc:1397] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:42:35.608642 24466 raft_consensus.cc:348] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.608881 24466 raft_consensus.cc:374] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:35.608966 24466 raft_consensus.cc:729] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e8fbacf5f74e329afba21f842adf13, State: Initialized, Role: FOLLOWER
I20260927 18:42:35.609414 24466 consensus_queue.cc:260] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.610026 24187 heartbeater.cc:503] Master 127.18.235.190:39895 was elected leader, sending a full tablet report...
I20260927 18:42:35.610324 24466 ts_tablet_manager.cc:1428] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:35.611876 24468 raft_consensus.cc:348] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.612064 24468 raft_consensus.cc:374] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:35.612149 24468 raft_consensus.cc:729] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770df888cdd24c6ea908420fdc8938df, State: Initialized, Role: FOLLOWER
I20260927 18:42:35.612196 24467 raft_consensus.cc:348] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.612427 24467 raft_consensus.cc:374] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:35.612485 24467 raft_consensus.cc:729] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bc58a498904f08874532300d0e6e68, State: Initialized, Role: FOLLOWER
I20260927 18:42:35.612581 24468 consensus_queue.cc:260] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.612872 24467 consensus_queue.cc:260] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.613545 24468 ts_tablet_manager.cc:1428] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:42:35.613736 24467 ts_tablet_manager.cc:1428] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 18:42:35.613883 24449 heartbeater.cc:503] Master 127.18.235.190:39895 was elected leader, sending a full tablet report...
I20260927 18:42:35.614048 24318 heartbeater.cc:503] Master 127.18.235.190:39895 was elected leader, sending a full tablet report...
W20260927 18:42:35.644373 24319 tablet.cc:2369] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:35.739456 24188 tablet.cc:2369] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:35.801194 24450 tablet.cc:2369] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:35.806175 24473 raft_consensus.cc:482] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:35.806320 24473 raft_consensus.cc:504] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.807331 24473 leader_election.cc:288] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15bc58a498904f08874532300d0e6e68 (127.18.235.130:32973), 54e8fbacf5f74e329afba21f842adf13 (127.18.235.129:40507)
I20260927 18:42:35.808938 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.807657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:54509 (local address 127.18.235.129:40507)
0927 18:42:35.807771 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.807775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.807813 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:42:35.807858 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.807861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.807897 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.807974 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.808574 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.808579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.808581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.808594 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.808618 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.808628 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.808709 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.808783 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.808819 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:35.808938 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.807552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40507 (local address 127.18.235.131:54509)
0927 18:42:35.807666 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.807680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.807757 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.808000 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.808003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.808016 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.808420 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.808421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.808431 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.808454 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.808706 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.808709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.808712 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.808783 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.808817 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:42:35.809243 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.807783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:46353 (local address 127.18.235.130:32973)
0927 18:42:35.807911 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.807916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.808116 (+ 200us) server_negotiation.cc:407] Connection header received
0927 18:42:35.808194 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.808197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.808227 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.808287 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.808844 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.808847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.808850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.808861 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.808879 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.808887 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.808957 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.809059 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.809098 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:42:35.809525 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.807552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:32973 (local address 127.18.235.131:46353)
0927 18:42:35.808031 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.808044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.808121 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.808310 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.808313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.808326 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.808704 (+ 378us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.808705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.808714 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.808734 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.808949 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.808952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.808955 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.809382 (+ 427us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.809418 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":113,"thread_start_us":133,"threads_started":1}
I20260927 18:42:35.809514 24141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b7b0f14f4a48418ed4d4b4a538f0dd" candidate_uuid: "770df888cdd24c6ea908420fdc8938df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e8fbacf5f74e329afba21f842adf13" is_pre_election: true
I20260927 18:42:35.809804 24141 raft_consensus.cc:2454] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770df888cdd24c6ea908420fdc8938df in term 0.
I20260927 18:42:35.810093 24472 raft_consensus.cc:482] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:35.810061 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b7b0f14f4a48418ed4d4b4a538f0dd" candidate_uuid: "770df888cdd24c6ea908420fdc8938df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bc58a498904f08874532300d0e6e68" is_pre_election: true
I20260927 18:42:35.810180 24472 raft_consensus.cc:504] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.810256 24337 leader_election.cc:302] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e8fbacf5f74e329afba21f842adf13, 770df888cdd24c6ea908420fdc8938df; no voters:
I20260927 18:42:35.810318 24272 raft_consensus.cc:2454] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770df888cdd24c6ea908420fdc8938df in term 0.
I20260927 18:42:35.810523 24473 raft_consensus.cc:2781] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:35.810606 24473 raft_consensus.cc:482] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:35.810655 24473 raft_consensus.cc:3037] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:35.811213 24472 leader_election.cc:288] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15bc58a498904f08874532300d0e6e68 (127.18.235.130:32973), 770df888cdd24c6ea908420fdc8938df (127.18.235.131:44263)
I20260927 18:42:35.811925 24473 raft_consensus.cc:504] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.812252 24473 leader_election.cc:288] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [CANDIDATE]: Term 1 election: Requested vote from peers 15bc58a498904f08874532300d0e6e68 (127.18.235.130:32973), 54e8fbacf5f74e329afba21f842adf13 (127.18.235.129:40507)
I20260927 18:42:35.812536 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b7b0f14f4a48418ed4d4b4a538f0dd" candidate_uuid: "770df888cdd24c6ea908420fdc8938df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bc58a498904f08874532300d0e6e68"
I20260927 18:42:35.812639 24141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b7b0f14f4a48418ed4d4b4a538f0dd" candidate_uuid: "770df888cdd24c6ea908420fdc8938df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e8fbacf5f74e329afba21f842adf13"
I20260927 18:42:35.812659 24272 raft_consensus.cc:3037] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:35.812741 24141 raft_consensus.cc:3037] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:35.813495 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.811461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:46021 (local address 127.18.235.131:44263)
0927 18:42:35.811603 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.811607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.812009 (+ 402us) server_negotiation.cc:407] Connection header received
0927 18:42:35.812137 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.812140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.812170 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.812232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.813099 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.813121 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.813124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.813135 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.813152 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.813162 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.813236 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.813365 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.813399 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:42:35.813686 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.811496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:53433 (local address 127.18.235.130:32973)
0927 18:42:35.811586 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.811589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.811911 (+ 322us) server_negotiation.cc:407] Connection header received
0927 18:42:35.812001 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.812004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.812034 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.812094 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.813317 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.813320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.813323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.813333 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.813350 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.813359 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.813409 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.813550 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.813582 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:42:35.813935 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.811350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:44263 (local address 127.18.235.129:46021)
0927 18:42:35.811935 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.811948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.812058 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.812311 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.812314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.812327 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.812953 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.812955 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.812965 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.812992 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.813273 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.813276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.813279 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.813763 (+ 484us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.813799 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":270,"thread_start_us":91,"threads_started":1}
I20260927 18:42:35.813938 24272 raft_consensus.cc:2454] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770df888cdd24c6ea908420fdc8938df in term 1.
I20260927 18:42:35.813925 24141 raft_consensus.cc:2454] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770df888cdd24c6ea908420fdc8938df in term 1.
I20260927 18:42:35.814182 24339 leader_election.cc:302] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bc58a498904f08874532300d0e6e68, 770df888cdd24c6ea908420fdc8938df; no voters:
I20260927 18:42:35.814126 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.811344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:32973 (local address 127.18.235.129:53433)
0927 18:42:35.811841 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.811852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:35.811924 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:35.812172 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:35.812175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:35.812189 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:35.813179 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:42:35.813181 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:35.813190 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:42:35.813210 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:35.813466 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:35.813468 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:35.813470 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 18:42:35.813975 (+ 505us) client_negotiation.cc:240] Negotiation successful
0927 18:42:35.814010 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":69,"threads_started":1}
I20260927 18:42:35.814337 24473 raft_consensus.cc:2781] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:35.814512 24473 raft_consensus.cc:686] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 1 LEADER]: Becoming Leader. State: Replica: 770df888cdd24c6ea908420fdc8938df, State: Running, Role: LEADER
I20260927 18:42:35.814673 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b7b0f14f4a48418ed4d4b4a538f0dd" candidate_uuid: "54e8fbacf5f74e329afba21f842adf13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bc58a498904f08874532300d0e6e68" is_pre_election: true
I20260927 18:42:35.814818 24272 raft_consensus.cc:2379] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54e8fbacf5f74e329afba21f842adf13 in current term 1: Already voted for candidate 770df888cdd24c6ea908420fdc8938df in this term.
I20260927 18:42:35.814788 24473 consensus_queue.cc:237] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } }
I20260927 18:42:35.816222 24403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b7b0f14f4a48418ed4d4b4a538f0dd" candidate_uuid: "54e8fbacf5f74e329afba21f842adf13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770df888cdd24c6ea908420fdc8938df" is_pre_election: true
I20260927 18:42:35.816699 24076 leader_election.cc:302] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e8fbacf5f74e329afba21f842adf13; no voters: 15bc58a498904f08874532300d0e6e68, 770df888cdd24c6ea908420fdc8938df
I20260927 18:42:35.816906 24472 raft_consensus.cc:2726] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 18:42:35.817751 23999 catalog_manager.cc:5495] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df reported cstate change: term changed from 0 to 1, leader changed from <none> to 770df888cdd24c6ea908420fdc8938df (127.18.235.131). New cstate: current_term: 1 leader_uuid: "770df888cdd24c6ea908420fdc8938df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770df888cdd24c6ea908420fdc8938df" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 44263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:35.881616 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.880478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.18.235.131:44263)
0927 18:42:35.880585 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:35.880588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:35.880599 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:35.880665 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:35.880667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:35.880723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:35.880786 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:35.881343 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:35.881346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:35.881348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:35.881358 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:35.881375 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:35.881383 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:35.881448 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:35.881492 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:42:35.881522 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:42:35.886914 24272 raft_consensus.cc:1261] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 1 FOLLOWER]: Refusing update from remote peer 770df888cdd24c6ea908420fdc8938df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:35.886952 24141 raft_consensus.cc:1261] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 1 FOLLOWER]: Refusing update from remote peer 770df888cdd24c6ea908420fdc8938df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:35.887482 24473 consensus_queue.cc:1035] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bc58a498904f08874532300d0e6e68" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:35.887655 24478 consensus_queue.cc:1035] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e8fbacf5f74e329afba21f842adf13" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:35.895970 24482 mvcc.cc:205] Tried to move back new op lower bound from 7334029540906680320 to 7334029540619657216. Current Snapshot: MvccSnapshot[applied={T|T < 7334029540906680320}]
I20260927 18:42:35.896389 24483 mvcc.cc:205] Tried to move back new op lower bound from 7334029540906680320 to 7334029540619657216. Current Snapshot: MvccSnapshot[applied={T|T < 7334029540906680320 or (T in {7334029540906680320})}]
I20260927 18:42:35.900538 23999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:35.900681 23999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:35.902782 23999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 170caf16f42348c1bff8d90750faa4b1: Sending DeleteTablet for 3 replicas of tablet 47b7b0f14f4a48418ed4d4b4a538f0dd
I20260927 18:42:35.903402 24383 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:35 UTC) from {username='slave'} at 127.0.0.1:59408
I20260927 18:42:35.903402 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:35 UTC) from {username='slave'} at 127.0.0.1:55924
I20260927 18:42:35.903787 24490 tablet_replica.cc:330] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: stopping tablet replica
I20260927 18:42:35.903779 24121 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:35 UTC) from {username='slave'} at 127.0.0.1:59028
I20260927 18:42:35.904049 24490 raft_consensus.cc:2229] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:35.904134 24492 tablet_replica.cc:330] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13: stopping tablet replica
I20260927 18:42:35.904366 24490 raft_consensus.cc:2258] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:35.904425 24492 raft_consensus.cc:2229] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 54e8fbacf5f74e329afba21f842adf13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:35.904673 24491 tablet_replica.cc:330] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: stopping tablet replica
I20260927 18:42:35.904714 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24059
I20260927 18:42:35.904922 24491 raft_consensus.cc:2229] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:35.905202 24491 raft_consensus.cc:2258] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:35.905755 24490 ts_tablet_manager.cc:1905] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:35.908039 24491 ts_tablet_manager.cc:1905] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:35.908255 24490 ts_tablet_manager.cc:1918] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:35.908334 24490 log.cc:1196] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/47b7b0f14f4a48418ed4d4b4a538f0dd
I20260927 18:42:35.908610 24490 ts_tablet_manager.cc:1939] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 15bc58a498904f08874532300d0e6e68: Deleting consensus metadata
I20260927 18:42:35.909690 23986 catalog_manager.cc:4841] TS 15bc58a498904f08874532300d0e6e68 (127.18.235.130:32973): tablet 47b7b0f14f4a48418ed4d4b4a538f0dd (table test-table [id=7a48b6d6c7394b14b3b91500fe14401d]) successfully deleted
I20260927 18:42:35.910475 24491 ts_tablet_manager.cc:1918] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:35.910559 24491 log.cc:1196] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/47b7b0f14f4a48418ed4d4b4a538f0dd
I20260927 18:42:35.910897 24491 ts_tablet_manager.cc:1939] T 47b7b0f14f4a48418ed4d4b4a538f0dd P 770df888cdd24c6ea908420fdc8938df: Deleting consensus metadata
I20260927 18:42:35.911803 23985 catalog_manager.cc:4841] TS 770df888cdd24c6ea908420fdc8938df (127.18.235.131:44263): tablet 47b7b0f14f4a48418ed4d4b4a538f0dd (table test-table [id=7a48b6d6c7394b14b3b91500fe14401d]) successfully deleted
W20260927 18:42:35.913236 23984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:40507 (error 108)
I20260927 18:42:35.913372 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24190
I20260927 18:42:35.917874 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24321
I20260927 18:42:35.919657 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:35.919426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40507 (local address 127.0.0.1:59032)
0927 18:42:35.919531 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:35.919592 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:42:35.919783 23984 catalog_manager.cc:4584] TS 54e8fbacf5f74e329afba21f842adf13 (127.18.235.129:40507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47b7b0f14f4a48418ed4d4b4a538f0dd: Network error: Client connection negotiation failed: client connection to 127.18.235.129:40507: connect: Connection refused (error 111)
I20260927 18:42:35.922411 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 23968
2026-09-27T18:42:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1058 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 18:42:35.941396 19374 mini_oidc.cc:152] Starting JWKS server
I20260927 18:42:35.942785 19374 webserver.cc:460] Webserver started at https://127.0.0.1:40857/ using document root <none> and password file <none>
I20260927 18:42:35.942920 19374 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:42:35.943125 19374 webserver.cc:460] Webserver started at http://127.0.0.1:40391/ using document root <none> and password file <none>
2026-09-27T18:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:35Z Disabled control of system clock
I20260927 18:42:35.950616 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35283
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35283
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40857/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/testchainca.cert with env {}
W20260927 18:42:36.049942 24499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:36.050148 24499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:36.050184 24499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:36.053963 24499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:36.054004 24499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:36.054018 24499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:36.054032 24499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:36.054045 24499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:42:36.054062 24499 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:42:36.054080 24499 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40857/jwks.json
I20260927 18:42:36.058952 24499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40857/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:36.059722 24499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:36.060530 24499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:36.067754 24505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:36.067740 24504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.067926 24499 server_base.cc:1034] running on GCE node
W20260927 18:42:36.067734 24507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.068492 24499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:36.069475 24499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:36.070626 24499 hybrid_clock.cc:648] HybridClock initialized: now 1790534556070609 us; error 28 us; skew 500 ppm
I20260927 18:42:36.072218 24499 webserver.cc:460] Webserver started at http://127.18.235.190:46743/ using document root <none> and password file <none>
I20260927 18:42:36.072692 24499 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:36.072762 24499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:36.072942 24499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:36.074753 24499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "16a3cb5673ec4c90bc90308febbc1e97"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "0d7bb1d078c768593b21953b836a035f"
server_key_iv: "1586d52d778cfcf43c97806f7ca3e0bd"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.075237 24499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "16a3cb5673ec4c90bc90308febbc1e97"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "0d7bb1d078c768593b21953b836a035f"
server_key_iv: "1586d52d778cfcf43c97806f7ca3e0bd"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.078359 24499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:36.080358 24513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:36.081403 24499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:36.081501 24499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "16a3cb5673ec4c90bc90308febbc1e97"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "0d7bb1d078c768593b21953b836a035f"
server_key_iv: "1586d52d778cfcf43c97806f7ca3e0bd"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.081580 24499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:36.116840 24499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:36.117400 24499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:36.117544 24499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:36.123265 24499 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:35283
I20260927 18:42:36.123281 24565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:35283 every 8 connection(s)
I20260927 18:42:36.124040 24499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:36.124673 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24499
I20260927 18:42:36.124801 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:36.125062 19374 external_mini_cluster.cc:1428] Setting key 27519bfa52ed4273110bbf11a9402975
I20260927 18:42:36.127775 24566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:36.133597 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:36.126288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.18.235.190:35283)
0927 18:42:36.126802 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:36.126816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:36.126847 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:42:36.127426 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:36.127441 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:36.127788 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:36.128061 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:36.128363 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.129650 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.130802 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.131235 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.131337 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.132072 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:36.132077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:36.132088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:36.132428 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:36.132480 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:36.132499 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:36.132568 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:36.132716 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 18:42:36.132947 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260927 18:42:36.133617 24566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97: Bootstrap starting.
I20260927 18:42:36.135612 24566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:36.135931 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:42:36.136310 24566 log.cc:824] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:36.138181 24566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97: No bootstrap required, opened a new log
I20260927 18:42:36.141290 24566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a3cb5673ec4c90bc90308febbc1e97" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35283 } }
I20260927 18:42:36.141481 24566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:36.141520 24566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a3cb5673ec4c90bc90308febbc1e97, State: Initialized, Role: FOLLOWER
I20260927 18:42:36.141887 24566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a3cb5673ec4c90bc90308febbc1e97" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35283 } }
I20260927 18:42:36.142030 24566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:36.142092 24566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:36.142191 24566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:36.142925 24566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a3cb5673ec4c90bc90308febbc1e97" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35283 } }
I20260927 18:42:36.143182 24566 leader_election.cc:302] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a3cb5673ec4c90bc90308febbc1e97; no voters:
I20260927 18:42:36.143411 24566 leader_election.cc:288] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:36.143492 24571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:36.143690 24571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [term 1 LEADER]: Becoming Leader. State: Replica: 16a3cb5673ec4c90bc90308febbc1e97, State: Running, Role: LEADER
I20260927 18:42:36.143996 24571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a3cb5673ec4c90bc90308febbc1e97" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35283 } }
I20260927 18:42:36.144414 24566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:36.145371 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a3cb5673ec4c90bc90308febbc1e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a3cb5673ec4c90bc90308febbc1e97" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35283 } } }
I20260927 18:42:36.145522 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:36.145881 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a3cb5673ec4c90bc90308febbc1e97. Latest consensus state: current_term: 1 leader_uuid: "16a3cb5673ec4c90bc90308febbc1e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a3cb5673ec4c90bc90308febbc1e97" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35283 } } }
I20260927 18:42:36.146080 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:36.145864 24580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:36.148851 24580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:36.153492 24580 catalog_manager.cc:1347] Generated new cluster ID: 77847c536b954c2a90e159ef4673df43
I20260927 18:42:36.153548 24580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:36.192510 24580 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:36.193425 24580 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:36.197135 24580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16a3cb5673ec4c90bc90308febbc1e97: Generated new TSK 0
I20260927 18:42:36.197748 24580 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:42:36.222575 24596 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 18:42:36.222645 24596 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:42:36.223156 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:36.206433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.18.235.190:35283)
0927 18:42:36.206544 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:36.206548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:36.206709 (+ 161us) server_negotiation.cc:407] Connection header received
0927 18:42:36.206878 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:36.206881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:36.206932 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:36.207028 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:42:36.207213 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.208015 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.208787 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.209153 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.209246 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.209499 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:42:36.222825 (+ 13326us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:42:36.223042 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43260: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 18:42:36.223220 24569 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43260: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 18:42:36.224114 24595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:36.206342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35283 (local address 127.0.0.1:43260)
0927 18:42:36.206642 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:36.206653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:36.206749 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:36.207041 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:36.207043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:36.207052 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:42:36.207136 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:36.207140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.208096 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.208098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:36.208698 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:36.208705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.209249 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.209251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:36.209312 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.209352 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:42:36.222994 (+ 13642us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:42:36.223151 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.235.190:35283: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.235.190:35283: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40857/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 18:42:36.224740 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24499
2026-09-27T18:42:36Z chronyd exiting
I20260927 18:42:36.260685 19374 test_util.cc:175] -----------------------------------------------
I20260927 18:42:36.260754 19374 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534532155427-19374-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 18:42:36.261714 19374 mini_oidc.cc:152] Starting JWKS server
I20260927 18:42:36.262475 19374 webserver.cc:460] Webserver started at https://127.0.0.1:36035/ using document root <none> and password file <none>
I20260927 18:42:36.262553 19374 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:42:36.262754 19374 webserver.cc:460] Webserver started at http://127.0.0.1:39583/ using document root <none> and password file <none>
2026-09-27T18:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:36Z Disabled control of system clock
I20260927 18:42:36.270315 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:46397
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:46397
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/testchainca.cert with env {}
W20260927 18:42:36.370303 24602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:36.370512 24602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:36.370553 24602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:36.374379 24602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:36.374423 24602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:36.374436 24602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:36.374465 24602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:36.374490 24602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:42:36.374517 24602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:42:36.374543 24602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36035/jwks.json
I20260927 18:42:36.379467 24602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:46397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:36.380277 24602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:36.381137 24602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:36.388496 24607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:36.388504 24608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.388659 24602 server_base.cc:1034] running on GCE node
W20260927 18:42:36.388851 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.389475 24602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:36.390735 24602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:36.391914 24602 hybrid_clock.cc:648] HybridClock initialized: now 1790534556391884 us; error 43 us; skew 500 ppm
I20260927 18:42:36.393805 24602 webserver.cc:460] Webserver started at http://127.18.235.190:45283/ using document root <none> and password file <none>
I20260927 18:42:36.394435 24602 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:36.394519 24602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:36.394734 24602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:36.397023 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "a0442e7b162d4beb954eef76543f567f"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "bd6b6bb2523a1c358c67f4d2b467cc6d"
server_key_iv: "5461f415ad49fa32402c744e68431b3f"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.397523 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "a0442e7b162d4beb954eef76543f567f"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "bd6b6bb2523a1c358c67f4d2b467cc6d"
server_key_iv: "5461f415ad49fa32402c744e68431b3f"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.400612 24602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:36.402668 24616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:36.403698 24602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:36.403800 24602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "a0442e7b162d4beb954eef76543f567f"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "bd6b6bb2523a1c358c67f4d2b467cc6d"
server_key_iv: "5461f415ad49fa32402c744e68431b3f"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.403890 24602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:36.449955 24602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:36.450552 24602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:36.450706 24602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:36.456799 24602 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:46397
I20260927 18:42:36.456816 24668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:46397 every 8 connection(s)
I20260927 18:42:36.457648 24602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:36.460444 24669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:36.464977 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24602
I20260927 18:42:36.465073 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:36.465350 19374 external_mini_cluster.cc:1428] Setting key 974141987810361fa64ddef89e4de647
I20260927 18:42:36.465533 24669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f: Bootstrap starting.
I20260927 18:42:36.467873 24669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:36.468545 24669 log.cc:824] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:36.470230 24669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f: No bootstrap required, opened a new log
I20260927 18:42:36.473474 24669 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0442e7b162d4beb954eef76543f567f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46397 } }
I20260927 18:42:36.473632 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:36.466615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.18.235.190:46397)
0927 18:42:36.467223 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:36.467235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:36.467267 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:42:36.467835 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:36.467851 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:36.468226 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:36.468504 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:36.468898 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.469919 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.470798 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.471233 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.471371 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.472082 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:36.472086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:36.472096 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:36.472425 (+ 329us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:36.472473 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:36.472491 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:36.472570 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:36.472691 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:42:36.472917 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":134,"threads_started":1}
I20260927 18:42:36.473677 24669 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:36.473726 24669 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0442e7b162d4beb954eef76543f567f, State: Initialized, Role: FOLLOWER
I20260927 18:42:36.474144 24669 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0442e7b162d4beb954eef76543f567f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46397 } }
I20260927 18:42:36.474291 24669 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:36.474359 24669 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:36.474444 24669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:36.475162 24669 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0442e7b162d4beb954eef76543f567f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46397 } }
I20260927 18:42:36.475490 24669 leader_election.cc:302] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0442e7b162d4beb954eef76543f567f; no voters:
I20260927 18:42:36.475766 24669 leader_election.cc:288] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:36.475843 24674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:36.475989 24674 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [term 1 LEADER]: Becoming Leader. State: Replica: a0442e7b162d4beb954eef76543f567f, State: Running, Role: LEADER
I20260927 18:42:36.476130 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:42:36.476257 24674 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0442e7b162d4beb954eef76543f567f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46397 } }
I20260927 18:42:36.476879 24669 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:36.477677 24676 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0442e7b162d4beb954eef76543f567f. Latest consensus state: current_term: 1 leader_uuid: "a0442e7b162d4beb954eef76543f567f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0442e7b162d4beb954eef76543f567f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46397 } } }
I20260927 18:42:36.477800 24676 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:36.478159 24683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:36.478111 24675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0442e7b162d4beb954eef76543f567f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0442e7b162d4beb954eef76543f567f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46397 } } }
I20260927 18:42:36.478257 24675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:36.480568 24683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:36.485347 24683 catalog_manager.cc:1347] Generated new cluster ID: 869947d514be4b89b0fcd8f1f5bf6c53
I20260927 18:42:36.485401 24683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:36.507649 24683 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:36.508508 24683 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:36.514956 24683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0442e7b162d4beb954eef76543f567f: Generated new TSK 0
I20260927 18:42:36.515503 24683 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:42:36.538589 24699 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 18:42:36.538693 24699 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:42:36.539088 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:36.524277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.18.235.190:46397)
0927 18:42:36.524391 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:36.524396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:36.524557 (+ 161us) server_negotiation.cc:407] Connection header received
0927 18:42:36.524700 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:36.524702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:36.524743 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:36.524818 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:42:36.525043 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.525578 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.526750 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.526952 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.527012 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.527286 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:42:36.538834 (+ 11548us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:42:36.539001 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35474: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 18:42:36.539131 24672 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35474: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 18:42:36.539497 24698 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:36.524222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:46397 (local address 127.0.0.1:35474)
0927 18:42:36.524478 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:36.524492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:36.524597 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:36.524846 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:36.524849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:36.524856 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:42:36.524962 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:36.524968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.525672 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.525675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:36.526654 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:36.526664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.527043 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.527047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:36.527132 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.527158 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:42:36.538970 (+ 11812us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:42:36.539118 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.235.190:46397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260927 18:42:36.540009 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24602
2026-09-27T18:42:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 18:42:36.578212 19374 mini_oidc.cc:152] Starting JWKS server
I20260927 18:42:36.579041 19374 webserver.cc:460] Webserver started at https://127.0.0.1:41293/ using document root <none> and password file <none>
I20260927 18:42:36.579119 19374 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:42:36.579315 19374 webserver.cc:460] Webserver started at http://127.0.0.1:37397/ using document root <none> and password file <none>
2026-09-27T18:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:36Z Disabled control of system clock
I20260927 18:42:36.587033 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38625
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:38625
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41293/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:42:36.687978 24705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:36.688187 24705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:36.688220 24705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:36.692054 24705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:36.692097 24705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:36.692111 24705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:36.692130 24705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:36.692147 24705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:42:36.692160 24705 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:42:36.692178 24705 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41293/jwks.json
I20260927 18:42:36.696635 24705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:38625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41293/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:36.697477 24705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:36.698621 24705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:36.705940 24713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:36.705976 24710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:36.705953 24711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.706116 24705 server_base.cc:1034] running on GCE node
I20260927 18:42:36.706841 24705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:36.707813 24705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:36.708971 24705 hybrid_clock.cc:648] HybridClock initialized: now 1790534556708955 us; error 26 us; skew 500 ppm
I20260927 18:42:36.710664 24705 webserver.cc:460] Webserver started at http://127.18.235.190:41417/ using document root <none> and password file <none>
I20260927 18:42:36.711251 24705 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:36.711318 24705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:36.711511 24705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:36.713436 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "f1778a5cf49d4a0bb81e06135e8f78b3"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "6a218267570f3ccdd49c6df8ae9cc9e1"
server_key_iv: "737c910440ba1fc1abc88c8952c583a1"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.713976 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "f1778a5cf49d4a0bb81e06135e8f78b3"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "6a218267570f3ccdd49c6df8ae9cc9e1"
server_key_iv: "737c910440ba1fc1abc88c8952c583a1"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.717463 24705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:36.719552 24719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:36.720595 24705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:36.720688 24705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "f1778a5cf49d4a0bb81e06135e8f78b3"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "6a218267570f3ccdd49c6df8ae9cc9e1"
server_key_iv: "737c910440ba1fc1abc88c8952c583a1"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.720775 24705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:36.748018 24705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:36.748564 24705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:36.748705 24705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:36.754654 24705 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:38625
I20260927 18:42:36.754670 24771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:38625 every 8 connection(s)
I20260927 18:42:36.755463 24705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:36.757937 24772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:36.761432 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24705
I20260927 18:42:36.761521 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:36.761731 19374 external_mini_cluster.cc:1428] Setting key 400ba84d7d2516e7feb647d284b6e3cb
I20260927 18:42:36.763013 24772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3: Bootstrap starting.
I20260927 18:42:36.765064 24772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:36.765722 24772 log.cc:824] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:36.767503 24772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3: No bootstrap required, opened a new log
I20260927 18:42:36.770150 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:36.762915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.18.235.190:38625)
0927 18:42:36.763444 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:36.763458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:36.763491 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:42:36.764062 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:36.764077 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:36.764430 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:36.764677 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:36.764989 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.765984 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.766894 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.767530 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.767668 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.768323 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:36.768329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:36.768352 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:36.768807 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:36.768866 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:36.768890 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:36.768981 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:36.769064 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 18:42:36.769328 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260927 18:42:36.771380 24772 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38625 } }
I20260927 18:42:36.771618 24772 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:36.771710 24772 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1778a5cf49d4a0bb81e06135e8f78b3, State: Initialized, Role: FOLLOWER
I20260927 18:42:36.772136 24772 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38625 } }
I20260927 18:42:36.772284 24772 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:36.772347 24772 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:36.772439 24772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:36.772684 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:42:36.773289 24772 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38625 } }
I20260927 18:42:36.773604 24772 leader_election.cc:302] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1778a5cf49d4a0bb81e06135e8f78b3; no voters:
I20260927 18:42:36.773912 24772 leader_election.cc:288] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:36.773993 24777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:36.774174 24777 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [term 1 LEADER]: Becoming Leader. State: Replica: f1778a5cf49d4a0bb81e06135e8f78b3, State: Running, Role: LEADER
I20260927 18:42:36.774501 24777 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38625 } }
I20260927 18:42:36.774983 24772 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:36.775930 24779 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1778a5cf49d4a0bb81e06135e8f78b3. Latest consensus state: current_term: 1 leader_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38625 } } }
I20260927 18:42:36.776046 24779 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:36.776383 24786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:36.776358 24778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1778a5cf49d4a0bb81e06135e8f78b3" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38625 } } }
I20260927 18:42:36.776510 24778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:36.779006 24786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:36.783643 24786 catalog_manager.cc:1347] Generated new cluster ID: a8a0f3c84f0a4ec3a8f8d86c1755d1a2
I20260927 18:42:36.783697 24786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:36.796685 24786 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:36.797485 24786 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:36.801426 24786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1778a5cf49d4a0bb81e06135e8f78b3: Generated new TSK 0
I20260927 18:42:36.801934 24786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:42:36.817425 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:36.813009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.18.235.190:38625)
0927 18:42:36.813155 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:36.813159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:36.813298 (+ 139us) server_negotiation.cc:407] Connection header received
0927 18:42:36.813434 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:36.813436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:36.813477 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:36.813552 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:42:36.813762 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.814257 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.815445 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.815643 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.815705 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.815976 (+ 271us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:42:36.817082 (+ 1106us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:42:36.817304 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 18:42:36.817474 24775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 18:42:36.819304 24801 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:36.812947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38625 (local address 127.0.0.1:36274)
0927 18:42:36.813221 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:36.813236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:36.813339 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:36.813567 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:36.813570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:36.813577 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:42:36.813683 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:36.813689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.814346 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.814350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:36.815343 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:36.815353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:36.815730 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:36.815734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:36.815821 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:36.815856 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:42:36.817260 (+ 1404us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:42:36.819144 (+ 1884us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.235.190:38625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1706,"thread_start_us":73,"threads_started":1}
I20260927 18:42:36.820077 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24705
E20260927 18:42:36.830274 24802 webserver.cc:533] Webserver: sslize failed:
E20260927 18:42:36.830328 24802 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T18:42:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T18:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:36Z Disabled control of system clock
I20260927 18:42:36.866117 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:43505
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:43505
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/testchainca.cert with env {}
W20260927 18:42:36.968750 24806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:36.969053 24806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:36.969141 24806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:36.973097 24806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:36.973186 24806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:36.973203 24806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:36.973230 24806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:36.973253 24806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:42:36.973276 24806 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:42:36.973297 24806 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 18:42:36.978273 24806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:43505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:43505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:36.979099 24806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:36.979943 24806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:36.987357 24811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:36.987411 24812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.987586 24806 server_base.cc:1034] running on GCE node
W20260927 18:42:36.987375 24814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:36.988122 24806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:36.989082 24806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:36.990284 24806 hybrid_clock.cc:648] HybridClock initialized: now 1790534556990267 us; error 33 us; skew 500 ppm
I20260927 18:42:36.991848 24806 webserver.cc:460] Webserver started at http://127.18.235.190:35121/ using document root <none> and password file <none>
I20260927 18:42:36.992352 24806 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:36.992421 24806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:36.992606 24806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:36.994472 24806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "269f6cb6634344a22c15ab79d483f39f"
server_key_iv: "670fae5bd3ae5287dad2b02e0bbfe43e"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.994990 24806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "269f6cb6634344a22c15ab79d483f39f"
server_key_iv: "670fae5bd3ae5287dad2b02e0bbfe43e"
server_key_version: "encryptionkey@0"
I20260927 18:42:36.998294 24806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:42:37.000404 24820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:37.001470 24806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:37.001569 24806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0"
format_stamp: "Formatted at 2026-09-27 18:42:36 on dist-test-slave-6lr6"
server_key: "269f6cb6634344a22c15ab79d483f39f"
server_key_iv: "670fae5bd3ae5287dad2b02e0bbfe43e"
server_key_version: "encryptionkey@0"
I20260927 18:42:37.001650 24806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:37.026790 24806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:37.027297 24806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:37.027443 24806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:37.033370 24806 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:43505
I20260927 18:42:37.033604 24872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:43505 every 8 connection(s)
I20260927 18:42:37.034157 24806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:37.036561 24873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:37.040343 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24806
I20260927 18:42:37.040442 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:37.040663 19374 external_mini_cluster.cc:1428] Setting key 0cb5469c49696e88063f8153fea9d9b5
I20260927 18:42:37.041473 24873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0: Bootstrap starting.
I20260927 18:42:37.043507 24873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:37.044147 24873 log.cc:824] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:37.045748 24873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0: No bootstrap required, opened a new log
I20260927 18:42:37.048605 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:37.041802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.18.235.190:43505)
0927 18:42:37.042355 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:37.042367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:37.042400 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:42:37.042954 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:37.042968 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:37.043307 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:37.043584 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:37.043966 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:37.045015 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:37.045847 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:37.046278 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:37.046423 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:37.047158 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:37.047163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:37.047173 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:42:37.047503 (+ 330us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:37.047552 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:42:37.047570 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:37.047647 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:37.047773 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:42:37.047988 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260927 18:42:37.049052 24873 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43505 } }
I20260927 18:42:37.049312 24873 raft_consensus.cc:374] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:37.049365 24873 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1c7b9ca20c4a9cbc1c8a91f73e48d0, State: Initialized, Role: FOLLOWER
I20260927 18:42:37.049746 24873 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43505 } }
I20260927 18:42:37.049866 24873 raft_consensus.cc:388] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:37.049925 24873 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:37.050014 24873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:37.050848 24873 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43505 } }
I20260927 18:42:37.051043 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:42:37.051148 24873 leader_election.cc:302] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1c7b9ca20c4a9cbc1c8a91f73e48d0; no voters:
I20260927 18:42:37.051410 24873 leader_election.cc:288] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:37.051479 24878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:37.051627 24878 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [term 1 LEADER]: Becoming Leader. State: Replica: af1c7b9ca20c4a9cbc1c8a91f73e48d0, State: Running, Role: LEADER
I20260927 18:42:37.051961 24878 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43505 } }
I20260927 18:42:37.052491 24873 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:37.053541 24879 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43505 } } }
I20260927 18:42:37.053574 24880 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1c7b9ca20c4a9cbc1c8a91f73e48d0. Latest consensus state: current_term: 1 leader_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1c7b9ca20c4a9cbc1c8a91f73e48d0" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43505 } } }
I20260927 18:42:37.053740 24880 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:37.053740 24879 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:37.054112 24887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:37.056582 24887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:37.061188 24887 catalog_manager.cc:1347] Generated new cluster ID: a90f990e7e8f4790a43b12e5933ea47a
I20260927 18:42:37.061244 24887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:37.068697 24887 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:37.069772 24887 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:37.080526 24887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1c7b9ca20c4a9cbc1c8a91f73e48d0: Generated new TSK 0
I20260927 18:42:37.081192 24887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:42:37.162626 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:37.088742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.18.235.190:43505)
0927 18:42:37.088833 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:37.088836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:37.089044 (+ 208us) server_negotiation.cc:407] Connection header received
0927 18:42:37.089177 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:37.089180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:37.089230 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:37.089321 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:42:37.089488 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:37.090249 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:37.091047 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:37.091393 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:37.091487 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:37.091692 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:42:37.162152 (+ 70460us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:42:37.162483 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44932: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 18:42:37.162700 24902 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:37.088678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43505 (local address 127.0.0.1:44932)
0927 18:42:37.088956 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:37.088967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:37.089052 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:37.089318 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:37.089320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:37.089325 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:42:37.089403 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:37.089407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:37.090356 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:37.090358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:37.090953 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:37.090959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:37.091488 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:37.091490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:37.091546 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:37.091565 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:42:37.162372 (+ 70807us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:42:37.162539 (+ 167us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.235.190:43505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
W20260927 18:42:37.162714 24876 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44932: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 18:42:37.163375 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24806
2026-09-27T18:42:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (321 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (113 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (114 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:37 dist-test-slave-6lr6 krb5kdc[24914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:37 dist-test-slave-6lr6 krb5kdc[24914](info): set up 2 sockets
Sep 27 18:42:37 dist-test-slave-6lr6 krb5kdc[24914](info): commencing operation
krb5kdc: starting...
W20260927 18:42:38.332944 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:38 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:38.439770 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39745
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:38.538852 24927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:38.539064 24927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:38.539100 24927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:38.542932 24927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:38.542976 24927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:38.542994 24927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:38.543011 24927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:38.543027 24927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:38.547823 24927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:38.548586 24927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:38.549443 24927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:42:38.556658 24927 server_base.cc:1034] running on GCE node
W20260927 18:42:38.556612 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:38.556555 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:38.556545 24935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:38.557359 24927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:38.558264 24927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:38.559397 24927 hybrid_clock.cc:648] HybridClock initialized: now 1790534558559379 us; error 32 us; skew 500 ppm
Sep 27 18:42:38 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534558, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:38.561780 24927 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:38.562515 24927 webserver.cc:460] Webserver started at http://127.18.235.190:39083/ using document root <none> and password file <none>
I20260927 18:42:38.562964 24927 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:38.563023 24927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:38.563189 24927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:38.564944 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761"
format_stamp: "Formatted at 2026-09-27 18:42:38 on dist-test-slave-6lr6"
server_key: "2cac1e5a81b220be59c629d171457684"
server_key_iv: "2344012c005d496279309f1a9a795095"
server_key_version: "encryptionkey@0"
I20260927 18:42:38.565464 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761"
format_stamp: "Formatted at 2026-09-27 18:42:38 on dist-test-slave-6lr6"
server_key: "2cac1e5a81b220be59c629d171457684"
server_key_iv: "2344012c005d496279309f1a9a795095"
server_key_version: "encryptionkey@0"
I20260927 18:42:38.568512 24927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:42:38.570580 24942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:38.571578 24927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:38.571677 24927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761"
format_stamp: "Formatted at 2026-09-27 18:42:38 on dist-test-slave-6lr6"
server_key: "2cac1e5a81b220be59c629d171457684"
server_key_iv: "2344012c005d496279309f1a9a795095"
server_key_version: "encryptionkey@0"
I20260927 18:42:38.571762 24927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:38.593572 24927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:38.596156 24927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:38.596300 24927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:38.602366 24927 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:39745
I20260927 18:42:38.602391 24994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:39745 every 8 connection(s)
I20260927 18:42:38.603155 24927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:38.604043 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 24927
I20260927 18:42:38.604156 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:38.604429 19374 external_mini_cluster.cc:1428] Setting key 06863470ab980a9473ec03fb5b6f5cae
I20260927 18:42:38.605988 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:38.611634 24995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761: Bootstrap starting.
Sep 27 18:42:38 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:38.613906 24995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:38.614564 24995 log.cc:824] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:38.616366 24995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761: No bootstrap required, opened a new log
I20260927 18:42:38.617053 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:38.605600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.18.235.190:39745)
0927 18:42:38.605957 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:38.605971 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:38.606001 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:42:38.606588 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:38.606604 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:38.606896 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:38.607142 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:38.607946 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:38.608876 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:38.610015 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:38.610444 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:38.610546 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:38.612976 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:38.612981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:38.612991 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:38.613015 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:38.614585 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:38.615035 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:38.615040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:38.615046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:38.615171 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:38.615447 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:38.615449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:38.615450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:38.615787 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:38.615920 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:38.616166 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 18:42:38.616399 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":114,"threads_started":1}
I20260927 18:42:38.619555 24995 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39745 } }
I20260927 18:42:38.619738 24995 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:38.619791 24995 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71b4ab5f8eb4ce4ac07c7d4c9e41761, State: Initialized, Role: FOLLOWER
I20260927 18:42:38.620147 24995 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39745 } }
I20260927 18:42:38.620260 24995 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:38.620319 24995 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:38.620415 24995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:38.621191 24995 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39745 } }
I20260927 18:42:38.621490 24995 leader_election.cc:302] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71b4ab5f8eb4ce4ac07c7d4c9e41761; no voters:
I20260927 18:42:38.621753 24995 leader_election.cc:288] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:38.621840 25000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:38.622022 25000 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [term 1 LEADER]: Becoming Leader. State: Replica: d71b4ab5f8eb4ce4ac07c7d4c9e41761, State: Running, Role: LEADER
I20260927 18:42:38.622298 25000 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39745 } }
I20260927 18:42:38.622745 24995 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:38.623637 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39745 } } }
I20260927 18:42:38.623750 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:38.624014 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71b4ab5f8eb4ce4ac07c7d4c9e41761. Latest consensus state: current_term: 1 leader_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71b4ab5f8eb4ce4ac07c7d4c9e41761" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39745 } } }
I20260927 18:42:38.624101 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:38.624171 25009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:38.626927 25009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:38.631311 25009 catalog_manager.cc:1347] Generated new cluster ID: b43a5d4cec0f4cbb8310d5e71be862fb
I20260927 18:42:38.631367 25009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:38.639859 25009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:38.640923 25009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:38.651064 25009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71b4ab5f8eb4ce4ac07c7d4c9e41761: Generated new TSK 0
I20260927 18:42:38.651676 25009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:38.678159 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39745
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:38.780313 25021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:38.780532 25021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:38.780584 25021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:38.784448 25021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:38.784499 25021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:38.784574 25021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:38.789489 25021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:38.790354 25021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:38.791242 25021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:38.799646 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:38.799642 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:38.799783 25021 server_base.cc:1034] running on GCE node
W20260927 18:42:38.799640 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:38.800158 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:38.800668 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:38.801802 25021 hybrid_clock.cc:648] HybridClock initialized: now 1790534558801791 us; error 28 us; skew 500 ppm
Sep 27 18:42:38 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534558, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:38.804147 25021 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:38.804864 25021 webserver.cc:460] Webserver started at http://127.18.235.129:45357/ using document root <none> and password file <none>
I20260927 18:42:38.805387 25021 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:38.805449 25021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:38.805612 25021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:38.807395 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "96f0b34ddcfb47d2bb02ce965f80de47"
format_stamp: "Formatted at 2026-09-27 18:42:38 on dist-test-slave-6lr6"
server_key: "b7335243421f3646b3d581dc5c688175"
server_key_iv: "32a56988f3cd4db4222e4907cc4b05ff"
server_key_version: "encryptionkey@0"
I20260927 18:42:38.807895 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "96f0b34ddcfb47d2bb02ce965f80de47"
format_stamp: "Formatted at 2026-09-27 18:42:38 on dist-test-slave-6lr6"
server_key: "b7335243421f3646b3d581dc5c688175"
server_key_iv: "32a56988f3cd4db4222e4907cc4b05ff"
server_key_version: "encryptionkey@0"
I20260927 18:42:38.811113 25021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:38.813200 25036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:38.814306 25021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:38.814411 25021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "96f0b34ddcfb47d2bb02ce965f80de47"
format_stamp: "Formatted at 2026-09-27 18:42:38 on dist-test-slave-6lr6"
server_key: "b7335243421f3646b3d581dc5c688175"
server_key_iv: "32a56988f3cd4db4222e4907cc4b05ff"
server_key_version: "encryptionkey@0"
I20260927 18:42:38.814499 25021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:38.838910 25021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:38.841394 25021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:38.841544 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:38.842046 25021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:38.842810 25021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:38.842854 25021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:38.842902 25021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:38.842931 25021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:38.850979 25021 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:43795
I20260927 18:42:38.851007 25149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:43795 every 8 connection(s)
I20260927 18:42:38.851750 25021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:38.853016 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25021
I20260927 18:42:38.853127 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:38.853426 19374 external_mini_cluster.cc:1428] Setting key 9d19786968351c6c99ffabf67642ab5f
Sep 27 18:42:38 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534558, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:42:38.864271 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:38.853338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:44003 (local address 127.18.235.190:39745)
0927 18:42:38.853480 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:38.853485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:38.854279 (+ 794us) server_negotiation.cc:407] Connection header received
0927 18:42:38.855789 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:38.855791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:38.855832 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:38.855897 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:38.857083 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:38.857631 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:38.858731 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:38.858959 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:38.859041 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:38.861298 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:38.861301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:38.861303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:38.861325 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:38.862534 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:38.862996 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:38.862998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:38.863000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:38.863067 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:38.863361 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:38.863363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:38.863365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:38.863560 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:38.863632 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:38.864038 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 18:42:38.864147 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:38.865012 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:38.853687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39745 (local address 127.18.235.129:44003)
0927 18:42:38.854158 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:38.854196 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:38.855571 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:38.856066 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:38.856074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:38.856505 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:38.856953 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:38.856966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:38.857750 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:38.857753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:38.858627 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:38.858636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:38.859078 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:38.859082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:38.859236 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:38.859689 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:38.859703 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:38.861146 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:38.862671 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:38.862676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:38.862686 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:38.862911 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:38.863139 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:38.863141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:38.863143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:38.863264 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:38.863630 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:38.863634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:38.863844 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 18:42:38.864006 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:42:38.864292 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260927 18:42:38.866021 25150 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39745
I20260927 18:42:38.866226 25150 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:38.866616 25150 heartbeater.cc:511] Master 127.18.235.190:39745 requested a full tablet report, sending...
I20260927 18:42:38.868110 24959 ts_manager.cc:194] Registered new tserver with Master: 96f0b34ddcfb47d2bb02ce965f80de47 (127.18.235.129:43795)
I20260927 18:42:38.869632 24959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:44003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:38.877763 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39745
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:38.977406 25155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:38.977622 25155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:38.977685 25155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:38.981600 25155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:38.981660 25155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:38.981746 25155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:38.986644 25155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:38.987454 25155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:38.988302 25155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:38.996667 25161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:38.996670 25163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:38.996927 25155 server_base.cc:1034] running on GCE node
W20260927 18:42:38.997054 25160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:38.997522 25155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:38.998220 25155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:38.999385 25155 hybrid_clock.cc:648] HybridClock initialized: now 1790534558999372 us; error 26 us; skew 500 ppm
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.001902 25155 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:39.002696 25155 webserver.cc:460] Webserver started at http://127.18.235.130:46307/ using document root <none> and password file <none>
I20260927 18:42:39.003175 25155 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:39.003235 25155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:39.003418 25155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:39.005245 25155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "4d755b02ac274e57a6dbe209c8d2ea3b"
format_stamp: "Formatted at 2026-09-27 18:42:39 on dist-test-slave-6lr6"
server_key: "c8e8b49ada03f335248624721efd148a"
server_key_iv: "4cc510aec4bd876d956dc13e01c2424a"
server_key_version: "encryptionkey@0"
I20260927 18:42:39.005752 25155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "4d755b02ac274e57a6dbe209c8d2ea3b"
format_stamp: "Formatted at 2026-09-27 18:42:39 on dist-test-slave-6lr6"
server_key: "c8e8b49ada03f335248624721efd148a"
server_key_iv: "4cc510aec4bd876d956dc13e01c2424a"
server_key_version: "encryptionkey@0"
I20260927 18:42:39.009068 25155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:42:39.011270 25170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:39.012351 25155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:39.012458 25155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "4d755b02ac274e57a6dbe209c8d2ea3b"
format_stamp: "Formatted at 2026-09-27 18:42:39 on dist-test-slave-6lr6"
server_key: "c8e8b49ada03f335248624721efd148a"
server_key_iv: "4cc510aec4bd876d956dc13e01c2424a"
server_key_version: "encryptionkey@0"
I20260927 18:42:39.012548 25155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:39.028283 25155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:39.030745 25155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:39.030908 25155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:39.031421 25155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:39.032177 25155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:39.032223 25155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:39.032267 25155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:39.032300 25155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:39.040772 25155 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42249
I20260927 18:42:39.040798 25283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42249 every 8 connection(s)
I20260927 18:42:39.041571 25155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:39.041953 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25155
I20260927 18:42:39.042071 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:39.042335 19374 external_mini_cluster.cc:1428] Setting key e2c29eb0f029d91f0eac0e5834d73ea0
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:42:39.055145 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.043243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:38819 (local address 127.18.235.190:39745)
0927 18:42:39.043343 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.043347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.044330 (+ 983us) server_negotiation.cc:407] Connection header received
0927 18:42:39.045835 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.045838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.045889 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.045958 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.047204 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.047776 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.049381 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.049598 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.049684 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.052105 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:39.052108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:39.052110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:39.052144 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:39.053423 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.053854 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.053857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.053859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.053930 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.054209 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.054212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.054214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.054380 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:39.054459 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:39.054883 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:42:39.054994 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:42:39.055806 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.043710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39745 (local address 127.18.235.130:38819)
0927 18:42:39.044160 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:39.044225 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:39.045652 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:39.046123 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:39.046129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:39.046496 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:39.047061 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:39.047078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.047899 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.047903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:39.049272 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:39.049284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.049735 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.049739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:39.049921 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.050443 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:39.050463 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:39.051946 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:39.053524 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:39.053529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:39.053542 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:39.053765 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:39.054006 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:39.054009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:39.054011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:39.054127 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:39.054470 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:39.054476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:39.054678 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 18:42:39.054826 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 18:42:39.055110 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
I20260927 18:42:39.056741 25284 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39745
I20260927 18:42:39.056963 25284 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:39.057487 25284 heartbeater.cc:511] Master 127.18.235.190:39745 requested a full tablet report, sending...
I20260927 18:42:39.058480 24959 ts_manager.cc:194] Registered new tserver with Master: 4d755b02ac274e57a6dbe209c8d2ea3b (127.18.235.130:42249)
I20260927 18:42:39.059000 24959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:38819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:39.067229 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39745
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:39.167514 25289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:39.167725 25289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:39.167768 25289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:39.171629 25289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:39.171672 25289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:39.171746 25289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:39.176678 25289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:39.177536 25289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:39.178401 25289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:39.186731 25297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:39.186731 25294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:39.186746 25295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:39.186815 25289 server_base.cc:1034] running on GCE node
I20260927 18:42:39.187355 25289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:39.187875 25289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:39.189018 25289 hybrid_clock.cc:648] HybridClock initialized: now 1790534559189003 us; error 32 us; skew 500 ppm
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.191521 25289 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:39.192261 25289 webserver.cc:460] Webserver started at http://127.18.235.131:36175/ using document root <none> and password file <none>
I20260927 18:42:39.192732 25289 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:39.192792 25289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:39.192957 25289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:39.194833 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "82980e16cc3f460194e8f22e32a7ce03"
format_stamp: "Formatted at 2026-09-27 18:42:39 on dist-test-slave-6lr6"
server_key: "9f2dc3910c128207036a71d580e47500"
server_key_iv: "a5e38d0527dc2246228617b3ed9b4c9f"
server_key_version: "encryptionkey@0"
I20260927 18:42:39.195343 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "82980e16cc3f460194e8f22e32a7ce03"
format_stamp: "Formatted at 2026-09-27 18:42:39 on dist-test-slave-6lr6"
server_key: "9f2dc3910c128207036a71d580e47500"
server_key_iv: "a5e38d0527dc2246228617b3ed9b4c9f"
server_key_version: "encryptionkey@0"
I20260927 18:42:39.198629 25289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:42:39.200721 25304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:39.201851 25289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:39.201947 25289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "82980e16cc3f460194e8f22e32a7ce03"
format_stamp: "Formatted at 2026-09-27 18:42:39 on dist-test-slave-6lr6"
server_key: "9f2dc3910c128207036a71d580e47500"
server_key_iv: "a5e38d0527dc2246228617b3ed9b4c9f"
server_key_version: "encryptionkey@0"
I20260927 18:42:39.202026 25289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:39.223847 25289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:39.226513 25289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:39.226660 25289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:39.227129 25289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:39.227823 25289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:39.227866 25289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:39.227912 25289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:39.227941 25289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:39.235683 25289 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:33845
I20260927 18:42:39.235706 25417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:33845 every 8 connection(s)
I20260927 18:42:39.236418 25289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:39.241982 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25289
I20260927 18:42:39.242066 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:39.242281 19374 external_mini_cluster.cc:1428] Setting key b507e9bb2638a82d29405bffaace5f2a
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:39.247978 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.237954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:38761 (local address 127.18.235.190:39745)
0927 18:42:39.238082 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.238087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.238811 (+ 724us) server_negotiation.cc:407] Connection header received
0927 18:42:39.239993 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.239995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.240039 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.240104 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.241067 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.241608 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.242725 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.243017 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.243108 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.245349 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:39.245352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:39.245354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:39.245377 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:39.246560 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.246923 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.246925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.246927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.246989 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.247229 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.247230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.247231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.247376 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:39.247440 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:39.247784 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 18:42:39.247868 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:39.248629 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.238294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39745 (local address 127.18.235.131:38761)
0927 18:42:39.238679 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:39.238715 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:39.239813 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:39.240238 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:39.240245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:39.240542 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:39.240962 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:39.240975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.241707 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.241710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:39.242619 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:39.242629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.243168 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.243172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:39.243312 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.243823 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:39.243842 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:39.245223 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:39.246635 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:39.246639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:39.246647 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:39.246856 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:39.247056 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:39.247058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:39.247059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:39.247164 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:39.247440 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:39.247444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:39.247624 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 18:42:39.247755 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:42:39.247957 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260927 18:42:39.249563 25418 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39745
I20260927 18:42:39.249744 25418 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:39.250119 25418 heartbeater.cc:511] Master 127.18.235.190:39745 requested a full tablet report, sending...
I20260927 18:42:39.250957 24959 ts_manager.cc:194] Registered new tserver with Master: 82980e16cc3f460194e8f22e32a7ce03 (127.18.235.131:33845)
I20260927 18:42:39.251458 24959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:38761
I20260927 18:42:39.255546 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/test-admin.keytab.
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.268736 19374 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 18:42:39.270953 25425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:39.269837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.18.235.129:43795)
0927 18:42:39.270109 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.270119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.270139 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:42:39.270180 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.270185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.270325 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.270436 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.270812 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42828: BlockingRecv error: recv got EOF from 127.0.0.1:42828 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260927 18:42:39.271045 19374 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.271905 19374 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.272650 19374 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:39.281193 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.273415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.18.235.129:43795)
0927 18:42:39.273523 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.273527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.273580 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:42:39.273698 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.273701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.273741 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.273799 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.274264 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.275074 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.275871 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.276225 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.276315 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.278231 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:39.278238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:39.278248 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:39.278271 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:39.279640 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.280015 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.280020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.280021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.280092 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.280320 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.280323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.280325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.280532 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:39.280713 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:39.280905 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 18:42:39.281034 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 18:42:39.284148 25425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:39.283463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.18.235.129:43795)
0927 18:42:39.283575 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.283578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.283631 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:42:39.283709 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.283711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.283741 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.283796 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.284059 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42840: BlockingRecv error: recv got EOF from 127.0.0.1:42840 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 18:42:39.284227 19374 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.285056 19374 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.285851 19374 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:39.292486 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.286480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.18.235.129:43795)
0927 18:42:39.286582 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.286585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.286656 (+ 71us) server_negotiation.cc:407] Connection header received
0927 18:42:39.286735 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.286737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.286767 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.286823 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.287183 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.287653 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.288298 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.288496 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.288558 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.290307 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:39.290310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:39.290311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:39.290330 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:39.291456 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.291747 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.291749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.291751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.291807 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.291962 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.291963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.291964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.292081 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:39.292155 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:39.292300 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 18:42:39.292392 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 18:42:39.294471 25425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:39.293813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.18.235.129:43795)
0927 18:42:39.293922 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.293926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.293981 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:42:39.294057 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.294058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.294086 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.294141 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.294393 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42858: BlockingRecv error: recv got EOF from 127.0.0.1:42858 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:42:39.294579 19374 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534532155427-19374-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.295378 19374 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:39.296106 19374 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:42:39.303051 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:39.296731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.18.235.129:43795)
0927 18:42:39.296834 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:39.296838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:39.296889 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:42:39.296995 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:39.296998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:39.297032 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:39.297093 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:39.297543 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.298005 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.298698 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:39.298917 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:39.298995 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:39.300736 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:39.300740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:39.300743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:39.300763 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:39.301778 (+ 1015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.302114 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.302117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.302119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.302182 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:39.302404 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:39.302407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:39.302409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:39.302539 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:39.302636 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:39.302833 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 18:42:39.302930 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:42:39.303821 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25021
I20260927 18:42:39.308935 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25155
I20260927 18:42:39.312951 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25289
I20260927 18:42:39.316905 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 24927
2026-09-27T18:42:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1926 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[25438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[25438](info): set up 2 sockets
Sep 27 18:42:39 dist-test-slave-6lr6 krb5kdc[25438](info): commencing operation
krb5kdc: starting...
W20260927 18:42:40.268978 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab.
Entry for principal oryx/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab.
I20260927 18:42:40.371531 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39097
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab
--principal=oryx/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:40.472666 25451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:40.472870 25451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:40.472905 25451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:40.476687 25451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:40.476733 25451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:40.476750 25451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:40.476765 25451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:40.476784 25451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:40.481658 25451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab
--principal=oryx/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:40.482452 25451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:40.483302 25451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:40.490808 25457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:40.490907 25451 server_base.cc:1034] running on GCE node
W20260927 18:42:40.490784 25459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:40.490875 25456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:40.491516 25451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:40.492450 25451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:40.493603 25451 hybrid_clock.cc:648] HybridClock initialized: now 1790534560493580 us; error 36 us; skew 500 ppm
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:40.495959 25451 init.cc:377] Logged in from keytab as oryx/127.18.235.190@KRBTEST.COM (short username oryx)
I20260927 18:42:40.496711 25451 webserver.cc:460] Webserver started at http://127.18.235.190:34273/ using document root <none> and password file <none>
I20260927 18:42:40.497212 25451 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:40.497277 25451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:40.497454 25451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:40.499238 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "4c5d48128c364721a4a3d3a537a3dc20"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "66aa810871dd1bb4262115dac031b080"
server_key_iv: "ee0ee74c26cf1f5fe58e2bf65e1e540b"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.499737 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "4c5d48128c364721a4a3d3a537a3dc20"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "66aa810871dd1bb4262115dac031b080"
server_key_iv: "ee0ee74c26cf1f5fe58e2bf65e1e540b"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.502952 25451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:40.505030 25466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.506163 25451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:40.506263 25451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "4c5d48128c364721a4a3d3a537a3dc20"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "66aa810871dd1bb4262115dac031b080"
server_key_iv: "ee0ee74c26cf1f5fe58e2bf65e1e540b"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.506346 25451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:40.519316 25451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:40.521555 25451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:40.521703 25451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:40.527570 25451 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:39097
I20260927 18:42:40.527797 25518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:39097 every 8 connection(s)
I20260927 18:42:40.528378 25451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:40.530799 25519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:40.535748 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Bootstrap starting.
I20260927 18:42:40.535821 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25451
I20260927 18:42:40.535936 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:40.536155 19374 external_mini_cluster.cc:1428] Setting key 4c80ab225bf7319e0c0b3ff0ea1b9aaa
I20260927 18:42:40.538111 25519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:40.538918 25519 log.cc:824] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:40.540808 25519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: No bootstrap required, opened a new log
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:40.544474 25519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } }
I20260927 18:42:40.544693 25519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:40.544749 25519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5d48128c364721a4a3d3a537a3dc20, State: Initialized, Role: FOLLOWER
I20260927 18:42:40.545270 25519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } }
I20260927 18:42:40.545428 25519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:40.545529 25519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:40.545624 25519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:40.546383 25519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } }
I20260927 18:42:40.546690 25519 leader_election.cc:302] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c5d48128c364721a4a3d3a537a3dc20; no voters:
I20260927 18:42:40.546957 25519 leader_election.cc:288] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:40.547034 25524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:40.547210 25524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 LEADER]: Becoming Leader. State: Replica: 4c5d48128c364721a4a3d3a537a3dc20, State: Running, Role: LEADER
I20260927 18:42:40.547499 25524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } }
I20260927 18:42:40.548177 25519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:40.548668 25524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } }
I20260927 18:42:40.548846 25524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:40.549273 25530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:40.549271 25525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5d48128c364721a4a3d3a537a3dc20. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } }
I20260927 18:42:40.549440 25525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:40.549717 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:40.537295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.18.235.190:39097)
0927 18:42:40.537663 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:40.537679 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:40.537717 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:42:40.538494 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:40.538516 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:40.538892 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:40.539228 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:40.539927 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.541343 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.542123 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.542800 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.542948 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:40.545163 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:40.545169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:40.545183 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:40.545212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:40.546834 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:40.547259 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:40.547265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:40.547271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:40.547397 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:40.547685 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:40.547688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:40.547706 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:40.548075 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:40.548269 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:40.548492 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:42:40.548781 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260927 18:42:40.553592 25530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:40.559173 25530 catalog_manager.cc:1347] Generated new cluster ID: 4735f9a81ed74473a2fdc18710c295db
I20260927 18:42:40.559243 25530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:40.568157 25530 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:40.569140 25530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:40.574537 25530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Generated new TSK 0
I20260927 18:42:40.575234 25530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab.
Entry for principal oryx/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab.
I20260927 18:42:40.608099 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39097
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:40.713881 25545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:40.714107 25545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:40.714155 25545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:40.718014 25545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:40.718062 25545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:40.718118 25545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:40.723013 25545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:40.723829 25545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:40.724689 25545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:40.734112 25553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:40.734628 25551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:40.734650 25550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:40.735240 25545 server_base.cc:1034] running on GCE node
I20260927 18:42:40.736040 25545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:40.736764 25545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:40.737978 25545 hybrid_clock.cc:648] HybridClock initialized: now 1790534560737934 us; error 66 us; skew 500 ppm
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:40.744050 25545 init.cc:377] Logged in from keytab as oryx/127.18.235.129@KRBTEST.COM (short username oryx)
I20260927 18:42:40.745082 25545 webserver.cc:460] Webserver started at http://127.18.235.129:34831/ using document root <none> and password file <none>
I20260927 18:42:40.745718 25545 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:40.745792 25545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:40.745999 25545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:40.748070 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "64db06eea6db4d66b6267908472f90f1"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "3b6a0b5a5481e8709540d567740446a0"
server_key_iv: "1bf02fc3c775bf7fe836222d50e19fe6"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.748646 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "64db06eea6db4d66b6267908472f90f1"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "3b6a0b5a5481e8709540d567740446a0"
server_key_iv: "1bf02fc3c775bf7fe836222d50e19fe6"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.752358 25545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:42:40.754771 25560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.755939 25545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:40.756035 25545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "64db06eea6db4d66b6267908472f90f1"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "3b6a0b5a5481e8709540d567740446a0"
server_key_iv: "1bf02fc3c775bf7fe836222d50e19fe6"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.756127 25545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:40.789449 25545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:40.792021 25545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:40.792177 25545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:40.792707 25545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:40.793501 25545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:40.793546 25545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.793591 25545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:40.793620 25545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.801563 25545 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:34049
I20260927 18:42:40.801656 25673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:34049 every 8 connection(s)
I20260927 18:42:40.802413 25545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:40.803980 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25545
I20260927 18:42:40.804091 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:40.804385 19374 external_mini_cluster.cc:1428] Setting key 114021707eabc25abf6aff4d5e2e6c8a
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.130@KRBTEST.COM" created.
I20260927 18:42:40.815526 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:40.804121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:35623 (local address 127.18.235.190:39097)
0927 18:42:40.804252 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:40.804255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:40.805267 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 18:42:40.806610 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:40.806613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:40.806663 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:40.806756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:40.807829 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.808413 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.809642 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.810035 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.810140 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:40.812375 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:40.812378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:40.812380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:40.812403 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:40.813688 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:40.814141 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:40.814144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:40.814145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:40.814215 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:40.814514 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:40.814517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:40.814518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:40.814687 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:40.814767 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:40.815230 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 18:42:40.815344 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:40.816332 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:40.804606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:35623)
0927 18:42:40.805078 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:40.805176 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:40.806409 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:40.806886 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:40.806893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:40.807230 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:40.807690 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:40.807703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.808506 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.808508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:40.809519 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:40.809529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.810166 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.810168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:40.810304 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:40.810846 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:40.810861 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:40.812217 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:40.813793 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:40.813799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:40.813812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:40.814051 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:40.814295 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:40.814298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:40.814300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:40.814430 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:40.814780 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:40.814787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:40.815017 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 18:42:40.815175 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 18:42:40.815480 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260927 18:42:40.817549 25674 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39097
I20260927 18:42:40.817751 25674 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:40.818177 25674 heartbeater.cc:511] Master 127.18.235.190:39097 requested a full tablet report, sending...
I20260927 18:42:40.819769 25483 ts_manager.cc:194] Registered new tserver with Master: 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049)
I20260927 18:42:40.821318 25483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:35623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab.
Entry for principal oryx/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab.
I20260927 18:42:40.829276 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39097
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:40.929332 25679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:40.929538 25679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:40.929577 25679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:40.933670 25679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:40.933714 25679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:40.933786 25679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:40.938603 25679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:40.939425 25679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:40.940284 25679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:40.948676 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:40.948688 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:40.948696 25679 server_base.cc:1034] running on GCE node
W20260927 18:42:40.948961 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:40.949360 25679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:40.949913 25679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:40.951046 25679 hybrid_clock.cc:648] HybridClock initialized: now 1790534560951021 us; error 39 us; skew 500 ppm
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:40.953379 25679 init.cc:377] Logged in from keytab as oryx/127.18.235.130@KRBTEST.COM (short username oryx)
I20260927 18:42:40.954077 25679 webserver.cc:460] Webserver started at http://127.18.235.130:42709/ using document root <none> and password file <none>
I20260927 18:42:40.954535 25679 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:40.954589 25679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:40.954763 25679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:40.956516 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "e632340f3f2049d7ad8beee6e97ba050"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "12c623fe10dd21b0bffd50a57bb714bd"
server_key_iv: "6bd20d61ec69e0c9bcbac039f03fdb1e"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.957000 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "e632340f3f2049d7ad8beee6e97ba050"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "12c623fe10dd21b0bffd50a57bb714bd"
server_key_iv: "6bd20d61ec69e0c9bcbac039f03fdb1e"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.960240 25679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:42:40.962334 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.963376 25679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:40.963461 25679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "e632340f3f2049d7ad8beee6e97ba050"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "12c623fe10dd21b0bffd50a57bb714bd"
server_key_iv: "6bd20d61ec69e0c9bcbac039f03fdb1e"
server_key_version: "encryptionkey@0"
I20260927 18:42:40.963547 25679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:40.977335 25679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:40.979676 25679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:40.979822 25679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:40.980304 25679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:40.981024 25679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:40.981066 25679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.981135 25679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:40.981165 25679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:40.989013 25679 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:35211
I20260927 18:42:40.989032 25807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:35211 every 8 connection(s)
I20260927 18:42:40.989776 25679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:40.993788 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25679
I20260927 18:42:40.993898 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:40.994172 19374 external_mini_cluster.cc:1428] Setting key 38ec09d43af70b9a95d77a8f519d3e97
Sep 27 18:42:40 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:42:41.002481 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:40.991260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:40487 (local address 127.18.235.190:39097)
0927 18:42:40.991423 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:40.991428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:40.992120 (+ 692us) server_negotiation.cc:407] Connection header received
0927 18:42:40.993326 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:40.993329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:40.993383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:40.993452 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:40.994747 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.995424 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.996500 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.996818 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.996914 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:40.999224 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:40.999227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:40.999230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:40.999257 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.000605 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.001095 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.001098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.001120 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.001219 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.001508 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.001512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.001514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.001683 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.001779 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.002215 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.002325 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:42:41.003172 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:40.991600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:40487)
0927 18:42:40.991983 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:40.992019 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:40.993149 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:40.993600 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:40.993608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:40.994037 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:40.994611 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:40.994626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.995550 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.995553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:40.996386 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:40.996395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:40.996970 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:40.996974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:40.997212 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:40.997720 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:40.997740 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:40.999079 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.000743 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.000748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.000761 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.001003 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.001299 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.001302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.001304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.001427 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.001775 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.001780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.001997 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.002148 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.002432 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260927 18:42:41.004192 25808 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39097
I20260927 18:42:41.004400 25808 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.131@KRBTEST.COM" created.
I20260927 18:42:41.004825 25808 heartbeater.cc:511] Master 127.18.235.190:39097 requested a full tablet report, sending...
I20260927 18:42:41.005818 25483 ts_manager.cc:194] Registered new tserver with Master: e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211)
I20260927 18:42:41.006493 25483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:40487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab.
Entry for principal oryx/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab.
I20260927 18:42:41.017903 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39097
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:41.120239 25813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:41.120466 25813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:41.120522 25813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:41.124425 25813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:41.124475 25813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:41.124549 25813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:41.129496 25813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:41.130383 25813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:41.131271 25813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:41.140354 25818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.140339 25819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.140328 25821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:41.140695 25813 server_base.cc:1034] running on GCE node
I20260927 18:42:41.141072 25813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:41.141708 25813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:41.142871 25813 hybrid_clock.cc:648] HybridClock initialized: now 1790534561142851 us; error 33 us; skew 500 ppm
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:41.145685 25813 init.cc:377] Logged in from keytab as oryx/127.18.235.131@KRBTEST.COM (short username oryx)
I20260927 18:42:41.146526 25813 webserver.cc:460] Webserver started at http://127.18.235.131:35499/ using document root <none> and password file <none>
I20260927 18:42:41.147022 25813 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.147084 25813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.147287 25813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:41.149294 25813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "a43588c46c044d5c9a72e29ce096646f"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "d3a58e99ba15dfa3ce930714af59f14c"
server_key_iv: "b002119303ca4e84da0a0da5274ab533"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.149848 25813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "a43588c46c044d5c9a72e29ce096646f"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "d3a58e99ba15dfa3ce930714af59f14c"
server_key_iv: "b002119303ca4e84da0a0da5274ab533"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.153343 25813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:41.155586 25828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.156656 25813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:41.156754 25813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "a43588c46c044d5c9a72e29ce096646f"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "d3a58e99ba15dfa3ce930714af59f14c"
server_key_iv: "b002119303ca4e84da0a0da5274ab533"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.156847 25813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.176337 25813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:41.178771 25813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:41.178920 25813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:41.179412 25813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:41.180137 25813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:41.180181 25813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.180233 25813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:41.180261 25813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.188120 25813 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:34875
I20260927 18:42:41.188169 25941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:34875 every 8 connection(s)
I20260927 18:42:41.188923 25813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:41.192524 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25813
I20260927 18:42:41.192611 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:41.192832 19374 external_mini_cluster.cc:1428] Setting key f98fa4b3903ff589e4b92d3e8573db66
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:41.200959 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.190568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:35293 (local address 127.18.235.190:39097)
0927 18:42:41.190688 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.190692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.191448 (+ 756us) server_negotiation.cc:407] Connection header received
0927 18:42:41.192755 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.192758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.192814 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.192884 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.194090 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.194614 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.195687 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.195907 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.195988 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.198121 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.198124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.198126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.198147 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.199226 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.199656 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.199660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.199661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.199733 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.200014 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.200017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.200018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.200188 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.200269 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.200697 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.200806 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:42:41.201689 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.190897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:35293)
0927 18:42:41.191312 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.191352 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.192536 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.193054 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.193060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.193449 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.193959 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.193972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.194727 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.194731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.195574 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.195583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.196028 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.196032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.196185 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.196643 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.196659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.197979 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.199322 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.199327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.199341 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.199563 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.199808 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.199811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.199814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.199933 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.200277 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.200283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.200491 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.200644 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.200931 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260927 18:42:41.202589 25942 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39097
I20260927 18:42:41.202790 25942 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:41.203212 25942 heartbeater.cc:511] Master 127.18.235.190:39097 requested a full tablet report, sending...
I20260927 18:42:41.204079 25483 ts_manager.cc:194] Registered new tserver with Master: a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875)
I20260927 18:42:41.204602 25483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:35293
I20260927 18:42:41.206137 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.189.keytab.
Entry for principal oryx/127.18.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.189.keytab.
I20260927 18:42:41.229185 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:44241
--webserver_interface=127.18.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.189.keytab
--principal=oryx/127.18.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39097,127.18.235.189:44241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:41.330252 25947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:41.330503 25947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:41.330549 25947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:41.334388 25947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:41.334434 25947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:41.334451 25947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:41.334468 25947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:41.334496 25947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:41.339381 25947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39097,127.18.235.189:44241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.189.keytab
--principal=oryx/127.18.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
W20260927 18:42:41.339540 25947 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.235.190:39097,127.18.235.189:44241'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 18:42:41.340267 25947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:41.341187 25947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:41.348667 25953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.348683 25955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.348683 25952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:41.349076 25947 server_base.cc:1034] running on GCE node
I20260927 18:42:41.349633 25947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:41.350594 25947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:41.351742 25947 hybrid_clock.cc:648] HybridClock initialized: now 1790534561351726 us; error 31 us; skew 500 ppm
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:41.354176 25947 init.cc:377] Logged in from keytab as oryx/127.18.235.189@KRBTEST.COM (short username oryx)
I20260927 18:42:41.354878 25947 webserver.cc:460] Webserver started at http://127.18.235.189:37077/ using document root <none> and password file <none>
I20260927 18:42:41.355326 25947 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.355376 25947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.355536 25947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:41.357328 25947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/instance:
uuid: "141243b2f105401f95d3938889172241"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "ef1a61201d93f39c9519077a6b175f15"
server_key_iv: "c179f31da8728ae0740eff81bba18620"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.357787 25947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal/instance:
uuid: "141243b2f105401f95d3938889172241"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "ef1a61201d93f39c9519077a6b175f15"
server_key_iv: "c179f31da8728ae0740eff81bba18620"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.361080 25947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:41.363229 25962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.364310 25947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:41.364398 25947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
uuid: "141243b2f105401f95d3938889172241"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "ef1a61201d93f39c9519077a6b175f15"
server_key_iv: "c179f31da8728ae0740eff81bba18620"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.364468 25947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.375485 25947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:41.377939 25947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:41.378063 25947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:41.384117 25947 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.189:44241
I20260927 18:42:41.384382 26014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.189:44241 every 8 connection(s)
I20260927 18:42:41.384941 25947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
I20260927 18:42:41.387485 26015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:41.391036 26015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:42:41.393527 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 25947
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.189@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
I20260927 18:42:41.404119 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.392696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.18.235.190:39097)
0927 18:42:41.392835 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.392839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.393545 (+ 706us) server_negotiation.cc:407] Connection header received
0927 18:42:41.394523 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.394526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.394578 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.394659 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.395750 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.396528 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.397768 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.398010 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.398105 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.400575 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.400579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.400582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.400607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.402149 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.402592 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.402596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.402597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.402686 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.403028 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.403030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.403031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.403308 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.403387 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.403850 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.403963 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:42:41.404865 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.392994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:39602)
0927 18:42:41.393410 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.393447 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.394278 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.394804 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.394810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.395156 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.395598 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.395622 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.396646 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.396649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.397663 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.397673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.398138 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.398141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.398270 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.398746 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.398761 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.400438 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.402246 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.402253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.402260 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.402500 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.402765 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.402768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.402771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.402945 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.403400 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.403406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.403622 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.403767 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.404040 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
I20260927 18:42:41.404877 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.394248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.18.235.189:44241)
0927 18:42:41.394558 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.394569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.394594 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:42:41.394783 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.394787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.394933 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.395057 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.395860 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.396930 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.397948 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.398369 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.398442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.401054 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.401061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.401072 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.401096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.402651 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.403116 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.403122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.403123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.403199 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.403452 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.403455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.403456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.403668 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.403835 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.404032 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.404153 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":10880,"thread_start_us":81,"threads_started":1}
I20260927 18:42:41.406611 26015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.189@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
I20260927 18:42:41.413019 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.406985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33884)
0927 18:42:41.407090 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.407104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.407176 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.407352 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.407353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.407471 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.407552 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.407557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.408197 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.408200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.408684 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.408692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.409123 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.409127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.409268 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.409662 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.409675 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.410786 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.412048 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.412050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.412051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.412200 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.412395 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.412397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.412398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.412460 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.412693 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.412695 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.412755 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.412817 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.412894 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":39}
I20260927 18:42:41.413045 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.407049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.18.235.189:44241)
0927 18:42:41.407135 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.407138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.407148 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:42:41.407243 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.407245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.407273 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.407330 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.407638 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.408099 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.408797 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.409000 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.409117 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.410901 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.410905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.410906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.410925 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.411975 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.412271 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.412273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.412274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.412334 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.412510 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.412512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.412513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.412636 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.412697 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.412856 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.412936 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":30}
I20260927 18:42:41.415787 26015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Bootstrap starting.
I20260927 18:42:41.417840 26015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:41.418483 26015 log.cc:824] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:41.420055 26015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: No bootstrap required, opened a new log
I20260927 18:42:41.422811 26015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } }
I20260927 18:42:41.423003 26015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:41.423056 26015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141243b2f105401f95d3938889172241, State: Initialized, Role: FOLLOWER
I20260927 18:42:41.423437 26015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } }
I20260927 18:42:41.423843 26021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } } }
I20260927 18:42:41.423949 26021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:41.424479 26015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:42:41.428728 26032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:41.428803 26032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:41.430765 25947 master_runner.cc:428] Detected that this master 141243b2f105401f95d3938889172241 is joining an existing cluster
I20260927 18:42:41.430822 25947 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.235.189:44241
I20260927 18:42:41.440598 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.433020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:39604)
0927 18:42:41.433296 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.433312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.433399 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.433606 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.433609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.433839 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.434005 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.434014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.434999 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.435003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.435663 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.435672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.436085 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.436089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.436173 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.436606 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.436619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.436927 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.438631 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.438634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.438636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.438865 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.439137 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.439141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.439143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.439224 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.440106 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.440109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.440192 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.440295 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.440416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":73,"thread_start_us":112,"threads_started":1}
I20260927 18:42:41.440635 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.432927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.18.235.190:39097)
0927 18:42:41.433037 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.433041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.433385 (+ 344us) server_negotiation.cc:407] Connection header received
0927 18:42:41.433474 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.433477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.433522 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.433593 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.434155 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.434927 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.435769 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.435983 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.436064 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.437060 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.437064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.437067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.437092 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.438526 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.438973 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.438977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.438979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.439064 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.439856 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.439859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.439860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.440018 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.440098 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.440375 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.440484 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:42:41.441310 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.432939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33898)
0927 18:42:41.433290 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.433310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.433410 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.433700 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.433702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.433880 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.433987 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.433992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.434965 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.434969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.435954 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.436016 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.436631 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.436635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.436718 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.437901 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.437916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.438299 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.439859 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.439863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.439864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.440093 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.440343 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.440346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.440348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.440421 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.440823 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.440825 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.440889 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.440971 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.441078 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":30,"thread_start_us":89,"threads_started":1}
I20260927 18:42:41.441332 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.432891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.18.235.189:44241)
0927 18:42:41.432991 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.432996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.433414 (+ 418us) server_negotiation.cc:407] Connection header received
0927 18:42:41.433511 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.433513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.433544 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.433601 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.434086 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.434835 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.436119 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.436446 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.436512 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.438429 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.438433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.438436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.438459 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.439766 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.440169 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.440172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.440174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.440262 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.440503 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.440506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.440507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.440698 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.440768 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.441045 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.441174 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":3328}
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.190@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
I20260927 18:42:41.453079 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.445031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.18.235.189:44241)
0927 18:42:41.445148 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.445151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.445452 (+ 301us) server_negotiation.cc:407] Connection header received
0927 18:42:41.445603 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.445606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.445643 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.445700 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.446407 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.446879 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.448050 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.448238 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.448303 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.450476 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.450479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.450481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.450500 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.451630 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.452031 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.452034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.452036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.452107 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.452339 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.452342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.452344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.452484 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.452560 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.452855 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.452947 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:42:41.453128 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.444993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33902)
0927 18:42:41.445318 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.445382 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.445501 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.445742 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.445748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.446148 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.446262 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.446270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.446968 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.446971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.447924 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.447934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.448323 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.448326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.448442 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.448873 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.448886 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.450362 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.451732 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.451740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.451746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.451948 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.452186 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.452189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.452191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.452265 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.452574 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.452580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.452680 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.452798 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.452953 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260927 18:42:41.454394 25483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.235.189:44241: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "141243b2f105401f95d3938889172241"
member_type: NON_VOTER
last_known_addr {
host: "127.18.235.189"
port: 44241
}
attrs {
promote: true
}
}
dest_uuid: "4c5d48128c364721a4a3d3a537a3dc20"
cas_config_opid_index: -1
I20260927 18:42:41.454982 25483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } }
I20260927 18:42:41.456501 26042 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 141243b2f105401f95d3938889172241 (127.18.235.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } }
I20260927 18:42:41.457078 25989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:41.458088 26042 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.235.189:44241
I20260927 18:42:41.458185 25989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Refusing update from remote peer 4c5d48128c364721a4a3d3a537a3dc20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 18:42:41.458455 25947 master.cc:537] Master@127.18.235.189:44241 shutting down...
I20260927 18:42:41.460096 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } }
I20260927 18:42:41.460194 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:41.460578 26042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:41.462225 25947 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:41.462468 25947 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:41.462564 25947 tablet_replica.cc:330] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: stopping tablet replica
W20260927 18:42:41.462759 25470 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 18:42:41.463474 25470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 -> Peer 141243b2f105401f95d3938889172241 (127.18.235.189:44241): Couldn't send request to peer 141243b2f105401f95d3938889172241. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.18.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.188@KRBTEST.COM" created.
I20260927 18:42:41.477190 25947 master.cc:552] Master@127.18.235.189:44241 shutdown complete.
I20260927 18:42:41.477236 25947 master_runner.cc:305] Clearing existing system tablet
I20260927 18:42:41.477458 25947 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.477512 25947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.478757 25947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.479470 26049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.479728 25947 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.479830 25947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.479866 25947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
uuid: "141243b2f105401f95d3938889172241"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "ef1a61201d93f39c9519077a6b175f15"
server_key_iv: "c179f31da8728ae0740eff81bba18620"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.481737 25947 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:41.484579 25947 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:42:41.484655 25947 log.cc:1196] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 18:42:41.484946 25947 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Deleting consensus metadata
I20260927 18:42:41.485090 25947 master_runner.cc:315] Copying system tablet from 127.18.235.190:39097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.188.keytab.
Entry for principal oryx/127.18.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.188.keytab.
I20260927 18:42:41.486146 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:46221
--webserver_interface=127.18.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.188.keytab
--principal=oryx/127.18.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:41.486222 25947 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: tablet copy: Beginning tablet copy session from remote peer at address 127.18.235.190:39097
I20260927 18:42:41.495523 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.487428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:39614)
0927 18:42:41.487760 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.487773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.487883 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.488132 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.488135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.488316 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.488463 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.488470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.489523 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.489526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.490415 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.490426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.491015 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.491018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.491117 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.491640 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.491654 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.492037 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.493951 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.493954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.493957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.494191 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.494476 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.494479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.494481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.494566 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.494973 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.494976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.495070 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.495174 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.495302 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":87,"threads_started":1}
I20260927 18:42:41.495535 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.487511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.18.235.190:39097)
0927 18:42:41.487632 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.487636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.487859 (+ 223us) server_negotiation.cc:407] Connection header received
0927 18:42:41.487964 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.487968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.488024 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.488111 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.488577 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.489419 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.490543 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.490903 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.491013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.492183 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.492187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.492198 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.492227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.493829 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.494288 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.494292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.494294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.494386 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.494653 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.494656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.494658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.494876 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.494980 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.495251 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.495362 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:42:41.496352 25503 tablet_copy_service.cc:140] P 4c5d48128c364721a4a3d3a537a3dc20: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 141243b2f105401f95d3938889172241 ({username='oryx', principal='oryx/127.18.235.189@KRBTEST.COM'} at 127.0.0.1:39614)
I20260927 18:42:41.496513 25503 tablet_copy_service.cc:161] P 4c5d48128c364721a4a3d3a537a3dc20: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 141243b2f105401f95d3938889172241 at {username='oryx', principal='oryx/127.18.235.189@KRBTEST.COM'} at 127.0.0.1:39614: session id = 141243b2f105401f95d3938889172241-00000000000000000000000000000000
I20260927 18:42:41.498207 25503 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:42:41.499630 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:42:41.501683 25947 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: tablet copy: Starting download of 0 data blocks...
I20260927 18:42:41.501847 25947 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: tablet copy: Starting download of 1 WAL segments...
I20260927 18:42:41.503412 25947 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:42:41.504935 25503 tablet_copy_service.cc:342] P 4c5d48128c364721a4a3d3a537a3dc20: Request end of tablet copy session 141243b2f105401f95d3938889172241-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.235.189@KRBTEST.COM'} at 127.0.0.1:39614
I20260927 18:42:41.505038 25503 tablet_copy_service.cc:433] P 4c5d48128c364721a4a3d3a537a3dc20: ending tablet copy session 141243b2f105401f95d3938889172241-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 141243b2f105401f95d3938889172241
I20260927 18:42:41.512652 25947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:41.514097 26057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.514178 26060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.514582 26058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:41.515045 25947 server_base.cc:1034] running on GCE node
I20260927 18:42:41.515187 25947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:41.515463 25947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:41.516600 25947 hybrid_clock.cc:648] HybridClock initialized: now 1790534561516590 us; error 26 us; skew 500 ppm
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:41.518018 25947 init.cc:377] Logged in from keytab as oryx/127.18.235.189@KRBTEST.COM (short username oryx)
I20260927 18:42:41.518416 25947 webserver.cc:460] Webserver started at http://127.18.235.189:34801/ using document root <none> and password file <none>
I20260927 18:42:41.518576 25947 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.518610 25947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.519809 25947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.520700 26067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.520957 25947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.521031 25947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
uuid: "141243b2f105401f95d3938889172241"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "ef1a61201d93f39c9519077a6b175f15"
server_key_iv: "c179f31da8728ae0740eff81bba18620"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.521144 25947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.535202 25947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:41.537184 25947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:41.541635 25947 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.189:44241
I20260927 18:42:41.542158 25947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
I20260927 18:42:41.545491 26119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.189:44241 every 8 connection(s)
I20260927 18:42:41.546746 26120 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:42:41.547542 26120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Bootstrap starting.
I20260927 18:42:41.555105 26120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:42:41.555341 26120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: Bootstrap complete.
I20260927 18:42:41.555770 26120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } }
I20260927 18:42:41.555884 26120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 141243b2f105401f95d3938889172241, State: Initialized, Role: LEARNER
I20260927 18:42:41.555979 26120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } }
I20260927 18:42:41.556267 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } }
I20260927 18:42:41.556385 26120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:41.556442 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:42:41.559445 26134 catalog_manager.cc:1259] Loaded cluster ID: 4735f9a81ed74473a2fdc18710c295db
I20260927 18:42:41.559489 26134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: loading cluster ID for follower catalog manager: success
I20260927 18:42:41.560941 26134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: acquiring CA information for follower catalog manager: success
I20260927 18:42:41.561993 26134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:42:41.619113 26055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:41.619318 26055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:41.619352 26055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:41.623208 26055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:41.623255 26055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:41.623271 26055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:41.623286 26055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:41.623304 26055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:41.628153 26055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.188.keytab
--principal=oryx/127.18.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:41.628959 26055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:41.629832 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:42:41.637215 26055 server_base.cc:1034] running on GCE node
W20260927 18:42:41.637097 26140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.637151 26142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.637130 26139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:41.637915 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:41.638903 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:41.640051 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790534561640037 us; error 26 us; skew 500 ppm
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:41.642464 26055 init.cc:377] Logged in from keytab as oryx/127.18.235.188@KRBTEST.COM (short username oryx)
I20260927 18:42:41.643213 26055 webserver.cc:460] Webserver started at http://127.18.235.188:42785/ using document root <none> and password file <none>
I20260927 18:42:41.643679 26055 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.643743 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.643921 26055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:41.645778 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/instance:
uuid: "393d86ba2f354a2b9dfc8e3d45e0714d"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "5c0d6860c8b66763dc4f015927c31e9a"
server_key_iv: "f2a26e3a7263a0d3648102fd5d387f7a"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.646274 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal/instance:
uuid: "393d86ba2f354a2b9dfc8e3d45e0714d"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "5c0d6860c8b66763dc4f015927c31e9a"
server_key_iv: "f2a26e3a7263a0d3648102fd5d387f7a"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.649673 26055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:41.651782 26149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.652834 26055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:41.652935 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
uuid: "393d86ba2f354a2b9dfc8e3d45e0714d"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "5c0d6860c8b66763dc4f015927c31e9a"
server_key_iv: "f2a26e3a7263a0d3648102fd5d387f7a"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.653017 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.673656 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:41.675951 26055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:41.676100 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:41.682097 26055 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.188:46221
I20260927 18:42:41.682334 26201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.188:46221 every 8 connection(s)
I20260927 18:42:41.682941 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
I20260927 18:42:41.685395 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:41.688676 26202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:42:41.692504 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 26055
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.188@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.235.188@KRBTEST.COM
I20260927 18:42:41.700989 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.690370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.18.235.190:39097)
0927 18:42:41.690522 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.690526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.691189 (+ 663us) server_negotiation.cc:407] Connection header received
0927 18:42:41.691969 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.691972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.692017 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.692086 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.693274 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.694026 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.695126 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.695337 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.695435 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.697852 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.697855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.697857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.697882 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.699136 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.699609 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.699612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.699613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.699687 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.699989 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.699990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.699991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.700152 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.700219 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.700741 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.700852 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:42:41.701931 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.690656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:39628)
0927 18:42:41.691053 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.691091 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.691805 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.692218 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.692224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.692552 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.693054 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.693074 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.694150 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.694152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.695017 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.695025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.695483 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.695486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.695633 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.696105 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.696120 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.697724 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.699236 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.699244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.699252 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.699525 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.699759 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.699762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.699764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.699913 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.700228 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.700233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.700508 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.700719 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.701022 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":29,"thread_start_us":103,"threads_started":1}
I20260927 18:42:41.703182 26202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:42:41.703359 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.693192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.18.235.188:46221)
0927 18:42:41.693452 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.693462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.693484 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:41.693523 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.693528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.693698 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.693843 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.694440 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.695176 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.695973 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.696297 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.696388 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.698968 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.698973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.698982 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.699002 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.701691 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.702102 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.702108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.702110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.702193 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.702444 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.702447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.702449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.702671 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.702870 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.703021 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.703149 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.188@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
I20260927 18:42:41.710502 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.703586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33914)
0927 18:42:41.703721 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.703731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.703784 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.704093 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.704096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.704295 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.704382 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.704387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.705449 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.705460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.705997 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.706006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.706457 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.706461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.706560 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.706927 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.706938 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.707993 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.709434 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.709437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.709438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.709629 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.709843 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.709845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.709846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.709916 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.710171 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.710173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.710251 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.710319 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.710407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:42:41.710574 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.703681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.18.235.189:44241)
0927 18:42:41.703915 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.703918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.703928 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:42:41.703962 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.703964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.703998 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.704067 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.704510 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.705239 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.706113 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.706349 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.706440 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.708116 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.708119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.708120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.708140 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.709339 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.709705 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.709707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.709708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.709786 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.709974 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.709976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.709977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.710112 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.710178 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.710367 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.710454 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260927 18:42:41.711155 26202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.188@KRBTEST.COM for oryx/127.18.235.188@KRBTEST.COM
I20260927 18:42:41.716946 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.711481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40042)
0927 18:42:41.711573 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.711585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.711642 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.711799 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.711800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.711909 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.711967 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.711971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.712567 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.712569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.713025 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.713030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.713428 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.713431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.713484 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.713820 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.713830 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.714765 (+ 935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.715955 (+ 1190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.715957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.715958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.716117 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.716309 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.716311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.716312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.716380 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.716618 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.716619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.716679 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.716745 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.716821 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:42:41.716969 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.711534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.18.235.188:46221)
0927 18:42:41.711620 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.711622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.711632 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:42:41.711694 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.711696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.711725 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.711783 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.712038 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.712493 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.713094 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.713351 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.713419 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.714871 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.714873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.714875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.714894 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.715873 (+ 979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.716182 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.716184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.716185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.716252 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.716435 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.716437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.716438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.716561 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.716622 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.716786 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.716860 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:42:41.719555 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Bootstrap starting.
I20260927 18:42:41.721570 26202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:41.722186 26202 log.cc:824] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:41.723714 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: No bootstrap required, opened a new log
I20260927 18:42:41.726420 26202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } }
I20260927 18:42:41.726598 26202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:41.726647 26202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 393d86ba2f354a2b9dfc8e3d45e0714d, State: Initialized, Role: FOLLOWER
I20260927 18:42:41.726994 26202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } }
I20260927 18:42:41.727346 26208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } } }
I20260927 18:42:41.727460 26208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:41.727950 26202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:42:41.731923 26219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:41.731981 26219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:41.734609 26055 master_runner.cc:428] Detected that this master 393d86ba2f354a2b9dfc8e3d45e0714d is joining an existing cluster
I20260927 18:42:41.734668 26055 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.235.188:46221
I20260927 18:42:41.744655 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.737026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.18.235.190:39097)
0927 18:42:41.737154 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.737159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.737172 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:41.737326 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.737329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.737375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.737453 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.738127 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.738688 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.739600 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.739949 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.740050 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.741137 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.741141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.741144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.741167 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.742687 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.743143 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.743146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.743148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.743233 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.743875 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.743878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.743880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.744075 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.744165 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.744404 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.744508 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:42:41.744675 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.736666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:39638)
0927 18:42:41.736914 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.736927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.737224 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.737693 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.737695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.737878 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.738011 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.738018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.738780 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.738783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.739487 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.739497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.740050 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.740054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.740135 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.740597 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.740611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.741000 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.742804 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.742807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.742809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.743056 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.743702 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.743705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.743706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.743786 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.744157 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.744160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.744244 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.744349 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.744519 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260927 18:42:41.748064 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.737613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.18.235.188:46221)
0927 18:42:41.737716 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.737720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.741334 (+ 3614us) server_negotiation.cc:407] Connection header received
0927 18:42:41.741422 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.741424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.741456 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.741521 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.741991 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.742433 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.743334 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.743531 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.743605 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.744658 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.744661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.744664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.744683 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.746491 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.746931 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.746934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.746935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.746999 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.747237 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.747241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.747243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.747413 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.747488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.747749 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.747918 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":48}
I20260927 18:42:41.748102 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.737163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40044)
0927 18:42:41.741256 (+ 4093us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.741266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.741350 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.741596 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.741598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.741785 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.741899 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.741906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.742521 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.742524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.743237 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.743244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.743612 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.743615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.743722 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.744143 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.744157 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.744547 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.746602 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.746605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.746607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.746838 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.747082 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.747084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.747086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.747154 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.747493 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.747496 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.747578 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.747708 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.747866 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4028,"mutex_wait_us":19,"thread_start_us":95,"threads_started":1}
I20260927 18:42:41.755810 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25545
I20260927 18:42:41.762356 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.737419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.18.235.189:44241)
0927 18:42:41.737524 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.737528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.737685 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:42:41.737812 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.737815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.737855 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.737921 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.738353 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.739107 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.739946 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.740267 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.740355 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.758855 (+ 18500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.758859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.758862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.758886 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.760576 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.761009 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.761012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.761014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.761093 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.761464 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.761468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.761470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.761681 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.761749 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.762106 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.762216 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:42:41.762643 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:34049
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:39097
--webserver_port=34831
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:41.763185 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.736709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33916)
0927 18:42:41.737616 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.737626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.737739 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.737943 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.737946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.738138 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.738240 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.738247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.739217 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.739220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.739842 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.739849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.740354 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.740357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.740438 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.758323 (+ 17885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.758337 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.758699 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.760674 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.760677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.760679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.760900 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.761276 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.761279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.761281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.761371 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.761851 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.761853 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.761939 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.762902 (+ 963us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.763032 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":71,"threads_started":1}
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534560, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.190@KRBTEST.COM for oryx/127.18.235.188@KRBTEST.COM
I20260927 18:42:41.769235 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.759807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40052)
0927 18:42:41.759917 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.759930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.759997 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.761149 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.761152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.761319 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.761409 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.761415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.762357 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.762360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.763198 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.763208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.763757 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.763761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.763861 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.764271 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.764284 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.765763 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.767654 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.767657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.767659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.767888 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.768185 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.768188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.768190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.768277 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.768727 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.768730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.768822 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.768923 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.769052 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:42:41.769908 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.760749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.18.235.188:46221)
0927 18:42:41.760841 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.760845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.760855 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:42:41.760910 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.760913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.760954 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.761018 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.761526 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.762240 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.763321 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.763641 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.763740 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.765879 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.765882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.765885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.765908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.767521 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.768000 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.768003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.768005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.768085 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.768403 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.768406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.768408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.768595 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.768682 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.769678 (+ 996us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.769778 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:42:41.770845 25483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.235.188:46221: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d"
member_type: NON_VOTER
last_known_addr {
host: "127.18.235.188"
port: 46221
}
attrs {
promote: true
}
}
dest_uuid: "4c5d48128c364721a4a3d3a537a3dc20"
cas_config_opid_index: 5
I20260927 18:42:41.771484 25483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } }
I20260927 18:42:41.772498 26043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.773940 26043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.235.188:46221
I20260927 18:42:41.774096 26043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.774192 26043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:41.775020 26055 master.cc:537] Master@127.18.235.188:46221 shutting down...
I20260927 18:42:41.775233 26176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:41.776619 26176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Refusing update from remote peer 4c5d48128c364721a4a3d3a537a3dc20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:42:41.777436 26042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:42:41.779357 25468 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 -> Peer 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188:46221): Couldn't send request to peer 393d86ba2f354a2b9dfc8e3d45e0714d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:42:41.780449 26055 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:41.780699 26055 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:41.780795 26055 tablet_replica.cc:330] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: stopping tablet replica
I20260927 18:42:41.785128 26055 master.cc:552] Master@127.18.235.188:46221 shutdown complete.
I20260927 18:42:41.785174 26055 master_runner.cc:305] Clearing existing system tablet
I20260927 18:42:41.785382 26055 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.785431 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.785408 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.774646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.18.235.189:44241)
0927 18:42:41.774747 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.774751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.779643 (+ 4892us) server_negotiation.cc:407] Connection header received
0927 18:42:41.779684 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.779687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.779751 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.779835 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:41.780213 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.781006 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.783502 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.784592 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.784693 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.784823 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.785155 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.785283 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:42:41.785926 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.774449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33926)
0927 18:42:41.779489 (+ 5040us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.779502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.779572 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.779978 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.779981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.780000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:41.780079 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.780086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.781227 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.781231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.783372 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.783384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.784881 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.784885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.784977 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.785021 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.785086 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.785816 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4980}
I20260927 18:42:41.786502 26094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 LEARNER]: Refusing update from remote peer 4c5d48128c364721a4a3d3a537a3dc20: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:42:41.786697 26055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:41.786880 26042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 18:42:41.787402 26230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.787549 26055 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.787638 26055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.787667 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
uuid: "393d86ba2f354a2b9dfc8e3d45e0714d"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "5c0d6860c8b66763dc4f015927c31e9a"
server_key_iv: "f2a26e3a7263a0d3648102fd5d387f7a"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.788586 26094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.789479 26055 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:41.790302 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5d48128c364721a4a3d3a537a3dc20. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.790416 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:42:41.790488 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: NON_VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: true } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.790541 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:42:41.791580 26055 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:42:41.791659 26055 log.cc:1196] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 18:42:41.791885 26055 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Deleting consensus metadata
I20260927 18:42:41.792037 26055 master_runner.cc:315] Copying system tablet from 127.18.235.190:39097
I20260927 18:42:41.793040 26055 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.235.190:39097
I20260927 18:42:41.798223 26042 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: attempting to promote NON_VOTER 141243b2f105401f95d3938889172241 to VOTER
I20260927 18:42:41.798612 26042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } }
I20260927 18:42:41.799880 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.799396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40062)
0927 18:42:41.799750 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.799813 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:46221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
I20260927 18:42:41.800275 26094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 LEARNER]: Refusing update from remote peer 4c5d48128c364721a4a3d3a537a3dc20: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:42:41.800617 26043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 18:42:41.801453 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.801241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40070)
0927 18:42:41.801356 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.801397 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:46221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 18:42:41.802104 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.794321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.18.235.190:39097)
0927 18:42:41.794438 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.794442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.794454 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:41.794485 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.794487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.794525 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.794596 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.795052 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.795818 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.796870 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.797228 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.797321 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.798526 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.798530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.798533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.798557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.800117 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.800722 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.800725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.800727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.800809 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.801170 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.801174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.801175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.801382 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.801459 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.801874 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.801980 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 18:42:41.802135 25468 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 -> Peer 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188:46221): Couldn't send request to peer 393d86ba2f354a2b9dfc8e3d45e0714d. Status: Network error: Client connection negotiation failed: client connection to 127.18.235.188:46221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:42:41.802835 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.793485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:39640)
0927 18:42:41.793693 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.793705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.793794 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.794639 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.794642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.794828 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.794944 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.794951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.795924 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.795927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.796765 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.796775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.797331 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.797335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.797421 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.797923 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.797938 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.798387 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.800307 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.800310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.800312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.800612 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.800960 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.800962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.800964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.801049 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.801597 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.801600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.801689 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.802530 (+ 841us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.802663 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260927 18:42:41.802768 26043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 141243b2f105401f95d3938889172241 (127.18.235.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.803355 25503 tablet_copy_service.cc:140] P 4c5d48128c364721a4a3d3a537a3dc20: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 393d86ba2f354a2b9dfc8e3d45e0714d ({username='oryx', principal='oryx/127.18.235.188@KRBTEST.COM'} at 127.0.0.1:39640)
I20260927 18:42:41.803436 25503 tablet_copy_service.cc:161] P 4c5d48128c364721a4a3d3a537a3dc20: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 393d86ba2f354a2b9dfc8e3d45e0714d at {username='oryx', principal='oryx/127.18.235.188@KRBTEST.COM'} at 127.0.0.1:39640: session id = 393d86ba2f354a2b9dfc8e3d45e0714d-00000000000000000000000000000000
I20260927 18:42:41.803992 26043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.804080 26043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:41.804189 26043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.804265 26043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:41.804375 25503 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:42:41.804729 26094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 141243b2f105401f95d3938889172241 (127.18.235.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.805429 26055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:42:41.806143 26232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.806237 26232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:41.807330 26055 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: tablet copy: Starting download of 0 data blocks...
I20260927 18:42:41.807482 26055 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: tablet copy: Starting download of 1 WAL segments...
I20260927 18:42:41.808748 26055 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:42:41.810274 25503 tablet_copy_service.cc:342] P 4c5d48128c364721a4a3d3a537a3dc20: Request end of tablet copy session 393d86ba2f354a2b9dfc8e3d45e0714d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.235.188@KRBTEST.COM'} at 127.0.0.1:39640
I20260927 18:42:41.810345 25503 tablet_copy_service.cc:433] P 4c5d48128c364721a4a3d3a537a3dc20: ending tablet copy session 393d86ba2f354a2b9dfc8e3d45e0714d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 393d86ba2f354a2b9dfc8e3d45e0714d
I20260927 18:42:41.816545 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:41.818081 26239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.821878 26240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.821944 26242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:41.822181 26055 server_base.cc:1034] running on GCE node
I20260927 18:42:41.822312 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:41.822568 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:41.823711 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790534561823699 us; error 27 us; skew 500 ppm
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:41.825142 26055 init.cc:377] Logged in from keytab as oryx/127.18.235.188@KRBTEST.COM (short username oryx)
I20260927 18:42:41.825521 26055 webserver.cc:460] Webserver started at http://127.18.235.188:40753/ using document root <none> and password file <none>
I20260927 18:42:41.825677 26055 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.825712 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.826807 26055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.827682 26249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.827935 26055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:41.828006 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
uuid: "393d86ba2f354a2b9dfc8e3d45e0714d"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "5c0d6860c8b66763dc4f015927c31e9a"
server_key_iv: "f2a26e3a7263a0d3648102fd5d387f7a"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.828071 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.843118 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:41.844967 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:41.849630 26055 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.188:46221
I20260927 18:42:41.849735 26301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.188:46221 every 8 connection(s)
I20260927 18:42:41.850149 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
I20260927 18:42:41.851725 26302 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:42:41.852716 26302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Bootstrap starting.
I20260927 18:42:41.863412 26302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:42:41.863674 26302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Bootstrap complete.
I20260927 18:42:41.864207 26302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } }
I20260927 18:42:41.864346 26302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 393d86ba2f354a2b9dfc8e3d45e0714d, State: Initialized, Role: LEARNER
I20260927 18:42:41.864459 26302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } }
I20260927 18:42:41.864784 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:41.864908 26302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:41.864977 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: This master's current role is: LEARNER
I20260927 18:42:41.868577 26316 catalog_manager.cc:1259] Loaded cluster ID: 4735f9a81ed74473a2fdc18710c295db
I20260927 18:42:41.868656 26316 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: loading cluster ID for follower catalog manager: success
I20260927 18:42:41.870711 26316 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: acquiring CA information for follower catalog manager: success
I20260927 18:42:41.871969 26316 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:42:41.890623 26228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:41.890820 26228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:41.890866 26228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:41.894716 26228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:41.894758 26228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:41.894824 26228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:41.899799 26228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:34049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=34831
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:41.900663 26228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:41.901585 26228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:41.910080 26324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.910140 26322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:41.910089 26321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:41.910315 26228 server_base.cc:1034] running on GCE node
I20260927 18:42:41.910745 26228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:41.911249 26228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:41.912385 26228 hybrid_clock.cc:648] HybridClock initialized: now 1790534561912364 us; error 38 us; skew 500 ppm
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:41.914772 26228 init.cc:377] Logged in from keytab as oryx/127.18.235.129@KRBTEST.COM (short username oryx)
I20260927 18:42:41.915526 26228 webserver.cc:460] Webserver started at http://127.18.235.129:34831/ using document root <none> and password file <none>
I20260927 18:42:41.916000 26228 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:41.916072 26228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:41.919456 26228 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:41.921270 26331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.922365 26228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:41.922462 26228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "64db06eea6db4d66b6267908472f90f1"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "3b6a0b5a5481e8709540d567740446a0"
server_key_iv: "1bf02fc3c775bf7fe836222d50e19fe6"
server_key_version: "encryptionkey@0"
I20260927 18:42:41.922916 26228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:41.938135 26228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:41.940950 26228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:41.941098 26228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:41.941613 26228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:41.942324 26228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:41.942368 26228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.942415 26228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:41.942443 26228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:41.950950 26228 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:34049
I20260927 18:42:41.950999 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:34049 every 8 connection(s)
I20260927 18:42:41.951782 26228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:41.958737 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 26228
I20260927 18:42:41.958904 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25679
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
Sep 27 18:42:41 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534561, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.188@KRBTEST.COM
I20260927 18:42:41.966516 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.954702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:50379 (local address 127.18.235.188:46221)
0927 18:42:41.955069 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.955079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.955603 (+ 524us) server_negotiation.cc:407] Connection header received
0927 18:42:41.955686 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.955689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.955734 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.955804 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.956872 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.957622 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959471 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.959807 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959880 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.962715 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.962728 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.962731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.962756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.964209 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.964770 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.964774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.964776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.964861 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.965307 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.965310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.965312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.965503 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.965568 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.966249 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.966352 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":71,"threads_started":1}
I20260927 18:42:41.966511 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.954705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:58401 (local address 127.18.235.189:44241)
0927 18:42:41.954838 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.954843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.954859 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:42:41.955267 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.955271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.955323 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.955403 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.957174 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.957921 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959133 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.959413 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959479 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.963050 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.963053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.963056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.963081 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.964424 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.964839 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.964842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.964844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.964914 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.965307 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.965310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.965312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.965503 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.965568 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.966240 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.966352 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:42:41.966775 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.954912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:52253 (local address 127.18.235.190:39097)
0927 18:42:41.955041 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:41.955046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:41.955265 (+ 219us) server_negotiation.cc:407] Connection header received
0927 18:42:41.955369 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:41.955372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:41.955434 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:41.955505 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:41.956812 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.957640 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959087 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.959397 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959467 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.962484 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:41.962488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:41.962491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:41.962518 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:41.964803 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.965357 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.965361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.965363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.965444 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:41.965808 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:41.965811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:41.965813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:41.966038 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:41.966103 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:41.966484 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 18:42:41.966585 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:42:41.967001 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:35211
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:39097
--webserver_port=42709
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:41.967900 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.953733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.18.235.129:50379)
0927 18:42:41.955524 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.955535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.955613 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.955856 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.955859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.956148 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.956683 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.956733 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.957755 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.957758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.959353 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.959361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.960042 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.960046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.960130 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.960591 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.960604 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.962595 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.964346 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.964351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.964366 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.964666 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.964967 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.964969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.964971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.965201 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.965645 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.965650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.965969 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.966681 (+ 712us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.966987 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":684,"spinlock_wait_cycles":16768,"thread_start_us":93,"threads_started":1}
I20260927 18:42:41.967897 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.953732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.18.235.129:58401)
0927 18:42:41.954208 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.954253 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.955074 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.955581 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.955590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.956149 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.956685 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.956702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.958017 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.958020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.959021 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.959031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.959640 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.959848 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.960354 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.960369 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.962902 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.964524 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.964527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.964529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.964746 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.965040 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.965043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.965045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.965201 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.965645 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.965650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.965969 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.966649 (+ 680us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.967046 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":279,"spinlock_wait_cycles":23936,"thread_start_us":133,"threads_started":1}
I20260927 18:42:41.969029 26445 heartbeater.cc:347] Connected to a master server at 127.18.235.188:46221
I20260927 18:42:41.969038 26447 heartbeater.cc:347] Connected to a master server at 127.18.235.189:44241
I20260927 18:42:41.969277 26447 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:41.969280 26445 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:41.969686 26445 heartbeater.cc:511] Master 127.18.235.188:46221 requested a full tablet report, sending...
I20260927 18:42:41.969683 26447 heartbeater.cc:511] Master 127.18.235.189:44241 requested a full tablet report, sending...
I20260927 18:42:41.970801 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:41.953733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:52253)
0927 18:42:41.955174 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:41.955205 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:41.955294 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:41.955611 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:41.955613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:41.956149 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:41.956685 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.956702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.957761 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.957764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.958963 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:41.958973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:41.959641 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:41.959644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:41.959852 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:41.960474 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:41.960490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:41.962289 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:41.964963 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.964966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.964968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.965252 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.965540 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:41.965584 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:41.965587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:41.965704 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:41.966230 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:41.966233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:41.966319 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:42:41.970499 (+ 4180us) client_negotiation.cc:240] Negotiation successful
0927 18:42:41.970628 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":266,"spinlock_wait_cycles":53120,"thread_start_us":117,"threads_started":1}
I20260927 18:42:41.971060 26075 ts_manager.cc:194] Registered new tserver with Master: 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049)
I20260927 18:42:41.971060 26264 ts_manager.cc:194] Registered new tserver with Master: 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049)
I20260927 18:42:41.971812 26446 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39097
I20260927 18:42:41.971882 26446 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:41.972023 26446 heartbeater.cc:511] Master 127.18.235.190:39097 requested a full tablet report, sending...
I20260927 18:42:41.972676 25483 ts_manager.cc:194] Re-registered known tserver with Master: 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049)
I20260927 18:42:41.973277 25483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:52253
W20260927 18:42:42.070876 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:42.071076 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:42.071120 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:42.074908 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:42.074951 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:42.075018 26453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:42.079800 26453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:35211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=42709
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:42.080597 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:42.081455 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:42.089689 26459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:42.089807 26453 server_base.cc:1034] running on GCE node
W20260927 18:42:42.089751 26458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:42.089756 26461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:42.090268 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:42.090798 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:42.091939 26453 hybrid_clock.cc:648] HybridClock initialized: now 1790534562091911 us; error 40 us; skew 500 ppm
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:42.094321 26453 init.cc:377] Logged in from keytab as oryx/127.18.235.130@KRBTEST.COM (short username oryx)
I20260927 18:42:42.095036 26453 webserver.cc:460] Webserver started at http://127.18.235.130:42709/ using document root <none> and password file <none>
I20260927 18:42:42.095504 26453 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:42.095577 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:42.099071 26453 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:42.100791 26468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:42.101890 26453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:42.101982 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "e632340f3f2049d7ad8beee6e97ba050"
format_stamp: "Formatted at 2026-09-27 18:42:40 on dist-test-slave-6lr6"
server_key: "12c623fe10dd21b0bffd50a57bb714bd"
server_key_iv: "6bd20d61ec69e0c9bcbac039f03fdb1e"
server_key_version: "encryptionkey@0"
I20260927 18:42:42.102453 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:42.120496 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:42.122932 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:42.123070 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:42.123533 26453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:42.124245 26453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:42.124289 26453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:42.124332 26453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:42.124358 26453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:42.132498 26453 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:35211
I20260927 18:42:42.132526 26581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:35211 every 8 connection(s)
I20260927 18:42:42.133347 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:42.142829 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 26453
I20260927 18:42:42.142982 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25813
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.188@KRBTEST.COM
I20260927 18:42:42.148391 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.134924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:44525 (local address 127.18.235.190:39097)
0927 18:42:42.135053 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.135058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.136018 (+ 960us) server_negotiation.cc:407] Connection header received
0927 18:42:42.136975 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.136978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.137026 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.137120 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.138441 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.138960 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140520 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.140857 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140943 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.143562 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.143566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.143568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.143598 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.145164 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.145793 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.145797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.145799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.146751 (+ 952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.147132 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.147136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.147138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.147476 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.147546 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.148123 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.148239 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:42.149128 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.135549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:37215 (local address 127.18.235.188:46221)
0927 18:42:42.135660 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.135664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.137222 (+ 1558us) server_negotiation.cc:407] Connection header received
0927 18:42:42.137332 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.137334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.137376 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.137442 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.138413 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.139228 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140927 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.141301 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.141381 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.145755 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.145759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.145762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.145788 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.147257 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.147845 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.147849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.147851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.147927 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.148249 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.148253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.148254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.148441 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.148509 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.148839 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.148964 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:42.149121 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:34875
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:39097
--webserver_port=35499
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:42.149770 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.135268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:44525)
0927 18:42:42.135886 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.135931 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.136778 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.137302 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.137310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.137813 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:42.138297 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.138303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.139101 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.139103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.140363 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.140372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.141043 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.141045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.141140 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.141590 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:42.141609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:42.143128 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:42.145280 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.145286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.145298 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.145674 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.146876 (+ 1202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.146879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.146881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.147034 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.147621 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:42.147627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:42.147881 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.148535 (+ 654us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.148857 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":113,"spinlock_wait_cycles":6400,"thread_start_us":144,"threads_started":1}
I20260927 18:42:42.149876 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.134926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:43421 (local address 127.18.235.189:44241)
0927 18:42:42.135065 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.135070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.136284 (+ 1214us) server_negotiation.cc:407] Connection header received
0927 18:42:42.136972 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.136976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.137026 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.137122 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.138456 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.139216 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140456 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.140767 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140864 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.145609 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.145613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.145615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.145640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.147167 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.147845 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.147849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.147850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.147927 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.148241 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.148244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.148246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.148450 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.148509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.149612 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.149721 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:42:42.149986 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.135457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.18.235.130:37215)
0927 18:42:42.137149 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.137161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.137256 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.137653 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.137656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.137837 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:42.138278 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.138291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.140183 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.140805 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.140813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.141527 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.141530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.141595 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.141934 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:42.141946 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:42.145621 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:42.147351 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.147353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.147355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.147737 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.148032 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.148035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.148037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.148150 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.148580 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:42.148583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:42.148663 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.149314 (+ 651us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.149437 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":17,"spinlock_wait_cycles":6272,"thread_start_us":87,"threads_started":1}
I20260927 18:42:42.150902 26583 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39097
I20260927 18:42:42.150946 26582 heartbeater.cc:347] Connected to a master server at 127.18.235.188:46221
I20260927 18:42:42.150949 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.135268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.18.235.130:43421)
0927 18:42:42.136185 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.136197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.136769 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.137361 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.137363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.137812 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:42.138294 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.138303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.139332 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.139335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.140356 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.140365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.140900 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.140904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.141074 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.141726 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:42.141748 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:42.145441 (+ 3693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:42.147325 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.147328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.147329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.147740 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.148033 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.148035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.148037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.148140 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.148988 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:42.148991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:42.149072 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.150662 (+ 1590us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.150799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":184,"spinlock_wait_cycles":60800,"thread_start_us":121,"threads_started":1}
I20260927 18:42:42.151115 26582 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:42.151587 26584 heartbeater.cc:347] Connected to a master server at 127.18.235.189:44241
I20260927 18:42:42.151639 26582 heartbeater.cc:511] Master 127.18.235.188:46221 requested a full tablet report, sending...
I20260927 18:42:42.151664 26584 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:42.151851 26584 heartbeater.cc:511] Master 127.18.235.189:44241 requested a full tablet report, sending...
I20260927 18:42:42.151114 26583 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:42.152212 26583 heartbeater.cc:511] Master 127.18.235.190:39097 requested a full tablet report, sending...
I20260927 18:42:42.152684 26075 ts_manager.cc:194] Registered new tserver with Master: e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211)
I20260927 18:42:42.152751 25483 ts_manager.cc:194] Re-registered known tserver with Master: e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211)
I20260927 18:42:42.152776 26264 ts_manager.cc:194] Registered new tserver with Master: e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211)
I20260927 18:42:42.153469 25483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:44525
W20260927 18:42:42.251705 26589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:42.251924 26589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:42.251977 26589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:42.255803 26589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:42.255849 26589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:42.255921 26589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:42.260802 26589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:34875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=35499
--tserver_master_addrs=127.18.235.190:39097,127.18.235.189:44241,127.18.235.188:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:42.261672 26589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:42.262547 26589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:42.270979 26594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:42.270989 26597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:42.271008 26595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:42.271169 26589 server_base.cc:1034] running on GCE node
I20260927 18:42:42.271602 26589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:42.272157 26589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:42.273304 26589 hybrid_clock.cc:648] HybridClock initialized: now 1790534562273290 us; error 29 us; skew 500 ppm
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:42.275813 26589 init.cc:377] Logged in from keytab as oryx/127.18.235.131@KRBTEST.COM (short username oryx)
I20260927 18:42:42.276546 26589 webserver.cc:460] Webserver started at http://127.18.235.131:35499/ using document root <none> and password file <none>
I20260927 18:42:42.277020 26589 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:42.277094 26589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:42.280483 26589 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 18:42:42.282321 26604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:42.283383 26589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:42.283479 26589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "a43588c46c044d5c9a72e29ce096646f"
format_stamp: "Formatted at 2026-09-27 18:42:41 on dist-test-slave-6lr6"
server_key: "d3a58e99ba15dfa3ce930714af59f14c"
server_key_iv: "b002119303ca4e84da0a0da5274ab533"
server_key_version: "encryptionkey@0"
I20260927 18:42:42.283950 26589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:42.305869 26589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:42.308318 26589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:42.308483 26589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:42.308975 26589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:42.309800 26589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:42.309846 26589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:42.309886 26589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:42.309913 26589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:42.317756 26589 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:34875
I20260927 18:42:42.317780 26717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:34875 every 8 connection(s)
I20260927 18:42:42.318601 26589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:42.325383 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 26589
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.189@KRBTEST.COM
Sep 27 18:42:42 dist-test-slave-6lr6 krb5kdc[25438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534562, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.188@KRBTEST.COM
I20260927 18:42:42.333950 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.320144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:34441 (local address 127.18.235.188:46221)
0927 18:42:42.320267 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.320272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.321577 (+ 1305us) server_negotiation.cc:407] Connection header received
0927 18:42:42.322427 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.322432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.322487 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.322563 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.323715 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.324500 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.325809 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.326144 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.326212 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.330052 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.330056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.330059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.330084 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.331574 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332124 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.332127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.332129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.332207 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332774 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.332778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.332780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.332974 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.333063 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.333681 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.333802 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:42:42.334159 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.320201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:41047 (local address 127.18.235.190:39097)
0927 18:42:42.320366 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.320370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.321575 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 18:42:42.322522 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.322525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.322574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.322644 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.323714 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.324213 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.325810 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.326144 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.326212 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.329769 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.329773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.329775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.329795 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.331938 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332394 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.332397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.332399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.332482 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332795 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.332798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.332800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.333001 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.333088 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.333912 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.334016 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:42:42.334797 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.320505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.18.235.131:34441)
0927 18:42:42.321458 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.321498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.322237 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.322741 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.322744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.323121 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:42.323580 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.323586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.324601 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.324604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.325683 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.325691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.327505 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.327511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.327602 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.328012 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:42.328025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:42.329659 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:42.331690 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.331697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.331710 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.332020 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.332302 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.332307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.332687 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.333148 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:42.333154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:42.333453 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.333620 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.333905 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":154,"spinlock_wait_cycles":77696,"thread_start_us":122,"threads_started":1}
I20260927 18:42:42.334810 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.320505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:41047)
0927 18:42:42.321457 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.321498 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.322268 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.322751 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.322754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.323128 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:42.323576 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.323586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.324355 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.324357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.325681 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.325691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.326392 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.326396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.326569 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.327072 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:42.327091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:42.329631 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:42.332045 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.332050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.332268 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.332574 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.332577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.332579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.332688 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.333152 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:42.333155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:42.333452 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.333619 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.334400 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":660,"spinlock_wait_cycles":64512,"thread_start_us":97,"threads_started":1}
I20260927 18:42:42.334851 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.320675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:54139 (local address 127.18.235.189:44241)
0927 18:42:42.320787 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.320791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.321584 (+ 793us) server_negotiation.cc:407] Connection header received
0927 18:42:42.322444 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.322447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.322492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.322539 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.323697 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.324495 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.326222 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.326539 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.326615 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.331644 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.331647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.331650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.331676 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.333198 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.333624 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.333627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.333629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.333707 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.334003 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.334006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.334008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.334195 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.334281 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.334598 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.334703 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:42:42.335029 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.320565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.18.235.131:54139)
0927 18:42:42.321456 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.321496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.322229 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.322727 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.322733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.323116 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:42.323566 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.323578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.325180 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.325183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.326081 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.326093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.326823 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.326840 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.327020 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.327456 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:42.327470 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:42.329996 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:42.333295 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.333298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.333300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.333514 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.333798 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:42.333801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:42.333803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:42.333904 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:42.334325 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:42.334328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:42.334409 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.334502 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.334604 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":97,"spinlock_wait_cycles":15488,"thread_start_us":89,"threads_started":1}
I20260927 18:42:42.335894 26719 heartbeater.cc:347] Connected to a master server at 127.18.235.190:39097
I20260927 18:42:42.336094 26719 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:42.336369 26720 heartbeater.cc:347] Connected to a master server at 127.18.235.189:44241
I20260927 18:42:42.336436 26720 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:42.336580 26720 heartbeater.cc:511] Master 127.18.235.189:44241 requested a full tablet report, sending...
I20260927 18:42:42.336580 26719 heartbeater.cc:511] Master 127.18.235.190:39097 requested a full tablet report, sending...
I20260927 18:42:42.337391 26718 heartbeater.cc:347] Connected to a master server at 127.18.235.188:46221
I20260927 18:42:42.337455 26718 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:42.337555 25483 ts_manager.cc:194] Re-registered known tserver with Master: a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875)
I20260927 18:42:42.337591 26718 heartbeater.cc:511] Master 127.18.235.188:46221 requested a full tablet report, sending...
I20260927 18:42:42.338181 25483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:41047
I20260927 18:42:42.338989 26264 ts_manager.cc:194] Registered new tserver with Master: a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875)
I20260927 18:42:42.339190 26075 ts_manager.cc:194] Registered new tserver with Master: a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875)
I20260927 18:42:42.341250 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.327144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.18.235.188:46221)
0927 18:42:42.327523 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.327526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.328580 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 18:42:42.328702 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.328704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.328750 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.328826 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:42.329070 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.329823 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.340050 (+ 10227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.340721 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.340839 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.340947 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.341063 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.341136 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":69,"threads_started":1}
I20260927 18:42:42.341274 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.327335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40080)
0927 18:42:42.327938 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.327951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.328623 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.328870 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.328873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.328892 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:42.328984 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.328991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.329920 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.329923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.331070 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.331081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.340848 (+ 9767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.340852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.340946 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.340970 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.341054 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.341150 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":61,"threads_started":1}
I20260927 18:42:42.341818 26270 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 18:42:42.342167 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5d48128c364721a4a3d3a537a3dc20. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:42.342272 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: This master's current role is: LEARNER
I20260927 18:42:42.343029 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.327049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.18.235.190:39097)
0927 18:42:42.332060 (+ 5011us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.332063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.332074 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:42.332401 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.332404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.332445 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.332548 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.335215 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.335971 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.337119 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.337414 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.337498 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.339034 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.339038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.339040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.339063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.340730 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.341889 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.341892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.341893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.341966 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.342239 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.342242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.342244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.342446 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.342564 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.342779 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.342882 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4925,"thread_start_us":77,"threads_started":1}
I20260927 18:42:42.343760 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.330565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.18.235.188:46221)
0927 18:42:42.331400 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.331413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.332331 (+ 918us) server_negotiation.cc:407] Connection header received
0927 18:42:42.332494 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.332497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.332537 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.332596 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.335954 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.336633 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.337680 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.337985 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.338050 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.339655 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.339658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.339661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.339683 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.341213 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.342361 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.342364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.342366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.342442 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.342705 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.342709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.342710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.342923 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.343037 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.343497 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.343598 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":58,"threads_started":1}
I20260927 18:42:42.345608 25483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:42.349009 25483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:42.349730 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.330805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.18.235.189:44241)
0927 18:42:42.331281 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.331285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.338630 (+ 7345us) server_negotiation.cc:407] Connection header received
0927 18:42:42.339982 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.339985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.340029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.340086 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.343311 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.344016 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.345237 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.345551 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.345654 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.346725 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.346729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.346732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.346755 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.348171 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.348560 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.348563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.348565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.348638 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.348878 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.348882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.348883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.349064 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.349244 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.349499 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.349596 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":68,"threads_started":1}
I20260927 18:42:42.351429 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: NON_VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: true } } }
I20260927 18:42:42.351544 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:42.354499 26740 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: attempting to promote NON_VOTER 393d86ba2f354a2b9dfc8e3d45e0714d to VOTER
I20260927 18:42:42.354832 26740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } }
I20260927 18:42:42.355928 26270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 LEARNER]: Refusing update from remote peer 4c5d48128c364721a4a3d3a537a3dc20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 18:42:42.355996 26094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Refusing update from remote peer 4c5d48128c364721a4a3d3a537a3dc20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 18:42:42.356273 26738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:42:42.356415 26042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 18:42:42.358229 26042 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.359405 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.359500 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:42.359617 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.359689 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:42.359771 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.359845 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:42.359776 26270 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.360406 26094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.361606 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.361737 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:42.362242 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c5d48128c364721a4a3d3a537a3dc20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:42.362715 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:42.368882 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.363819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:34875 (local address 127.0.0.1:44518)
0927 18:42:42.364027 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.364038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.364113 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.364662 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.364665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.364683 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:42.364819 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.364838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.366225 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.366228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.367525 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.367532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.368499 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.368502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.368587 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.368610 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.368679 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.368747 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":61,"spinlock_wait_cycles":3072,"thread_start_us":64,"threads_started":1}
I20260927 18:42:42.368919 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.363952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.18.235.131:34875)
0927 18:42:42.364220 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.364230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.364249 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:42:42.364308 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.364313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.364510 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.364976 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:42.365067 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.366098 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.367637 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.368377 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.368495 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.368606 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.368703 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.368795 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260927 18:42:42.369036 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.363547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:34049 (local address 127.0.0.1:52770)
0927 18:42:42.364023 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.364037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.364113 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.364792 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.364796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.364813 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:42.364891 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.364897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.366184 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.366187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.367505 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.367515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.368708 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.368712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.368795 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.368815 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.368864 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.368927 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":104,"spinlock_wait_cycles":4864}
I20260927 18:42:42.369355 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.364494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.18.235.130:35211)
0927 18:42:42.364902 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.364913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.365369 (+ 456us) server_negotiation.cc:407] Connection header received
0927 18:42:42.365452 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.365456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.365617 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.365749 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:42.365919 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.366732 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.367931 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.368486 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.368591 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.368703 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.369097 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.369222 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":91,"threads_started":1}
I20260927 18:42:42.369477 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.363698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.18.235.129:34049)
0927 18:42:42.364336 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.364348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.364368 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:42:42.364444 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.364448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.364626 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.365017 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:42.365142 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.366103 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.367637 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.368589 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.369175 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.369272 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.369308 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.369369 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":89,"threads_started":1}
I20260927 18:42:42.369598 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.364328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:35211 (local address 127.0.0.1:47894)
0927 18:42:42.365265 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.365278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.365365 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.365738 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.365741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.365757 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:42.365831 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.365836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.366842 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.366845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.367796 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.367805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.368758 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.368761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.368870 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.368893 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.368960 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.369031 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260927 18:42:42.372118 26379 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b (DEFAULT_TABLE table=test-table [id=f5039372fee04f40b28613eed22759fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:42.372131 26516 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b (DEFAULT_TABLE table=test-table [id=f5039372fee04f40b28613eed22759fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:42.372131 26652 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b (DEFAULT_TABLE table=test-table [id=f5039372fee04f40b28613eed22759fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:42.373029 26379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:42.373077 26516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:42.373077 26652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:42.379434 26754 tablet_bootstrap.cc:492] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Bootstrap starting.
I20260927 18:42:42.380605 26756 tablet_bootstrap.cc:492] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Bootstrap starting.
I20260927 18:42:42.381237 26755 tablet_bootstrap.cc:492] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Bootstrap starting.
I20260927 18:42:42.381894 26754 tablet_bootstrap.cc:654] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:42.382691 26756 tablet_bootstrap.cc:654] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:42.382781 26754 log.cc:824] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:42.383001 26755 tablet_bootstrap.cc:654] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:42.383438 26756 log.cc:824] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:42.383754 26755 log.cc:824] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:42.384780 26754 tablet_bootstrap.cc:492] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: No bootstrap required, opened a new log
I20260927 18:42:42.384979 26754 ts_tablet_manager.cc:1397] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:42:42.385304 26756 tablet_bootstrap.cc:492] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: No bootstrap required, opened a new log
I20260927 18:42:42.385486 26756 ts_tablet_manager.cc:1397] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:42.389446 26754 raft_consensus.cc:348] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.389708 26754 raft_consensus.cc:374] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:42.389761 26754 raft_consensus.cc:729] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64db06eea6db4d66b6267908472f90f1, State: Initialized, Role: FOLLOWER
I20260927 18:42:42.389621 26756 raft_consensus.cc:348] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.389822 26756 raft_consensus.cc:374] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:42.389874 26756 raft_consensus.cc:729] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43588c46c044d5c9a72e29ce096646f, State: Initialized, Role: FOLLOWER
I20260927 18:42:42.390214 26754 consensus_queue.cc:260] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.390214 26756 consensus_queue.cc:260] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.391175 26756 ts_tablet_manager.cc:1428] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:42:42.391175 26754 ts_tablet_manager.cc:1428] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:42:42.391522 26719 heartbeater.cc:503] Master 127.18.235.190:39097 was elected leader, sending a full tablet report...
I20260927 18:42:42.392531 26446 heartbeater.cc:503] Master 127.18.235.190:39097 was elected leader, sending a full tablet report...
I20260927 18:42:42.396466 26755 tablet_bootstrap.cc:492] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: No bootstrap required, opened a new log
I20260927 18:42:42.396653 26755 ts_tablet_manager.cc:1397] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260927 18:42:42.400089 26755 raft_consensus.cc:348] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.400276 26755 raft_consensus.cc:374] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:42.400312 26755 raft_consensus.cc:729] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632340f3f2049d7ad8beee6e97ba050, State: Initialized, Role: FOLLOWER
I20260927 18:42:42.400655 26755 consensus_queue.cc:260] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.401226 26583 heartbeater.cc:503] Master 127.18.235.190:39097 was elected leader, sending a full tablet report...
I20260927 18:42:42.401510 26755 ts_tablet_manager.cc:1428] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 18:42:42.453745 26448 tablet.cc:2369] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:42.497144 26762 raft_consensus.cc:482] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:42.497324 26762 raft_consensus.cc:504] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.498574 26762 leader_election.cc:288] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049), a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875)
I20260927 18:42:42.502570 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.498773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:34049 (local address 127.18.235.130:36871)
0927 18:42:42.498879 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.498893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.498977 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.499328 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.499331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.499344 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:42.499452 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.499459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.500371 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.500374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.501502 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.501512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.502225 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.502227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.502303 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.502323 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.502379 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.502430 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 18:42:42.502589 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.498930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:36871 (local address 127.18.235.129:34049)
0927 18:42:42.499142 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.499147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.499161 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:42:42.499205 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.499207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.499244 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.499297 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:42.499559 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.500273 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.501609 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.502118 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.502223 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.502274 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.502405 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.502475 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 18:42:42.502849 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.499022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:34875 (local address 127.18.235.130:52335)
0927 18:42:42.499121 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:42.499133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:42.499200 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:42.499832 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:42.499835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:42.499852 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:42.499938 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.499943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.500690 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.500693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.501733 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:42.501740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.502488 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.502492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:42.502586 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.502608 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:42:42.502675 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:42:42.502744 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 18:42:42.503014 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.499512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:52335 (local address 127.18.235.131:34875)
0927 18:42:42.499635 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.499640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.499652 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:42.499696 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.499699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.499741 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.499817 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:42.500025 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.500597 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.501825 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.502380 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.502484 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.502524 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.502823 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.502910 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:42.503098 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b" candidate_uuid: "e632340f3f2049d7ad8beee6e97ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64db06eea6db4d66b6267908472f90f1" is_pre_election: true
I20260927 18:42:42.503358 26399 raft_consensus.cc:2454] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632340f3f2049d7ad8beee6e97ba050 in term 0.
I20260927 18:42:42.503499 26672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b" candidate_uuid: "e632340f3f2049d7ad8beee6e97ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43588c46c044d5c9a72e29ce096646f" is_pre_election: true
I20260927 18:42:42.503768 26672 raft_consensus.cc:2454] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632340f3f2049d7ad8beee6e97ba050 in term 0.
I20260927 18:42:42.503799 26472 leader_election.cc:302] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64db06eea6db4d66b6267908472f90f1, e632340f3f2049d7ad8beee6e97ba050; no voters:
I20260927 18:42:42.504112 26762 raft_consensus.cc:2781] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:42.504185 26762 raft_consensus.cc:482] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:42.504235 26762 raft_consensus.cc:3037] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:42.505239 26762 raft_consensus.cc:504] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.505510 26762 leader_election.cc:288] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [CANDIDATE]: Term 1 election: Requested vote from peers 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049), a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875)
I20260927 18:42:42.505769 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b" candidate_uuid: "e632340f3f2049d7ad8beee6e97ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64db06eea6db4d66b6267908472f90f1"
I20260927 18:42:42.505860 26672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b" candidate_uuid: "e632340f3f2049d7ad8beee6e97ba050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43588c46c044d5c9a72e29ce096646f"
I20260927 18:42:42.505887 26399 raft_consensus.cc:3037] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:42.505963 26672 raft_consensus.cc:3037] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:42.507082 26672 raft_consensus.cc:2454] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632340f3f2049d7ad8beee6e97ba050 in term 1.
I20260927 18:42:42.507082 26399 raft_consensus.cc:2454] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632340f3f2049d7ad8beee6e97ba050 in term 1.
I20260927 18:42:42.507404 26472 leader_election.cc:302] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a43588c46c044d5c9a72e29ce096646f, e632340f3f2049d7ad8beee6e97ba050; no voters:
I20260927 18:42:42.507558 26762 raft_consensus.cc:2781] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:42.507768 26762 raft_consensus.cc:686] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 1 LEADER]: Becoming Leader. State: Replica: e632340f3f2049d7ad8beee6e97ba050, State: Running, Role: LEADER
I20260927 18:42:42.508062 26762 consensus_queue.cc:237] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:42.511154 25483 catalog_manager.cc:5495] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 reported cstate change: term changed from 0 to 1, leader changed from <none> to e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130). New cstate: current_term: 1 leader_uuid: "e632340f3f2049d7ad8beee6e97ba050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } health_report { overall_health: HEALTHY } } }
I20260927 18:42:42.550750 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.547071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.18.235.130:35211)
0927 18:42:42.547204 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.547207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.547220 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:42.547327 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.547329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.547378 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.547458 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:42.547658 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.548275 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.549447 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.549686 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.549790 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.550067 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:42.550423 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:42.550505 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.550578 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.550623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:42:42.556388 26672 raft_consensus.cc:1261] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Refusing update from remote peer e632340f3f2049d7ad8beee6e97ba050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:42.556402 26399 raft_consensus.cc:1261] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Refusing update from remote peer e632340f3f2049d7ad8beee6e97ba050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:42.557044 26762 consensus_queue.cc:1035] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:42.557260 26763 consensus_queue.cc:1035] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:42.568138 26768 mvcc.cc:205] Tried to move back new op lower bound from 7334029568223698944 to 7334029568035221504. Current Snapshot: MvccSnapshot[applied={T|T < 7334029568223698944}]
I20260927 18:42:42.569427 26766 mvcc.cc:205] Tried to move back new op lower bound from 7334029568223698944 to 7334029568035221504. Current Snapshot: MvccSnapshot[applied={T|T < 7334029568223698944 or (T in {7334029568223698944})}]
I20260927 18:42:42.571696 26767 mvcc.cc:205] Tried to move back new op lower bound from 7334029568223698944 to 7334029568035221504. Current Snapshot: MvccSnapshot[applied={T|T < 7334029568223698944 or (T in {7334029568223698944})}]
I20260927 18:42:42.574400 25482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:42.574533 25482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:42.577965 25482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c5d48128c364721a4a3d3a537a3dc20: Sending DeleteTablet for 3 replicas of tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b
I20260927 18:42:42.578688 26516 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:42 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:47894
I20260927 18:42:42.579195 26652 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:42 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:44518
I20260927 18:42:42.579218 26379 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:42 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:52770
I20260927 18:42:42.579638 26776 tablet_replica.cc:330] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: stopping tablet replica
I20260927 18:42:42.579921 26776 raft_consensus.cc:2229] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:42.580227 26776 raft_consensus.cc:2258] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:42.580571 26777 tablet_replica.cc:330] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: stopping tablet replica
I20260927 18:42:42.580832 26777 raft_consensus.cc:2229] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:42.581084 26777 raft_consensus.cc:2258] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:42.581677 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25451
I20260927 18:42:42.584100 26776 ts_tablet_manager.cc:1905] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:42.585290 26777 ts_tablet_manager.cc:1905] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:42.586663 26776 ts_tablet_manager.cc:1918] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:42.586750 26776 log.cc:1196] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b
I20260927 18:42:42.587033 26776 ts_tablet_manager.cc:1939] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P 64db06eea6db4d66b6267908472f90f1: Deleting consensus metadata
W20260927 18:42:42.588769 26332 connection.cc:535] server connection from 127.0.0.1:52770 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 18:42:42.589315 26068 connection.cc:535] server connection from 127.0.0.1:33926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:42:42.589723 26775 tablet_replica.cc:330] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: stopping tablet replica
I20260927 18:42:42.590000 26775 raft_consensus.cc:2229] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:42.590344 26775 raft_consensus.cc:2258] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:42.591593 26777 ts_tablet_manager.cc:1918] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:42.592626 26775 ts_tablet_manager.cc:1905] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:42.593400 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.580685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.18.235.189:44241)
0927 18:42:42.580811 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.580815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.580826 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:42.580859 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.580861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.580901 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.580965 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.582201 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.583003 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.586888 (+ 3885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.587275 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.587349 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.589522 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.589525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.589528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.589553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.591384 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.592063 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.592067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.592069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.592146 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.592429 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.592433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.592435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.592632 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.592737 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.593099 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.593249 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:42:42.594151 26777 log.cc:1196] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b
I20260927 18:42:42.594638 26777 ts_tablet_manager.cc:1939] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P a43588c46c044d5c9a72e29ce096646f: Deleting consensus metadata
I20260927 18:42:42.595595 26775 ts_tablet_manager.cc:1918] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:42.595690 26775 log.cc:1196] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b
I20260927 18:42:42.596009 26775 ts_tablet_manager.cc:1939] T cd8ee0ebaa2a4ea6ba4de5ef6c90fa4b P e632340f3f2049d7ad8beee6e97ba050: Deleting consensus metadata
W20260927 18:42:42.596748 26607 connection.cc:441] server connection from 127.0.0.1:44518 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44518 (request call id 1) could send its response
W20260927 18:42:42.597119 26470 connection.cc:441] server connection from 127.0.0.1:47894 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47894 (request call id 1) could send its response
I20260927 18:42:42.603152 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.591658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.18.235.189:44241)
0927 18:42:42.591753 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.591757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.592900 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 18:42:42.593072 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.593075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.593219 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.593269 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.594904 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.595632 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.597136 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.597445 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.597519 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.599558 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.599562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.599565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.599587 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.601494 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.601940 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.601944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.601945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.602029 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.602287 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.602291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.602292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.602480 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.602570 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.602912 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.603013 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:42:42.604590 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.581534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.18.235.188:46221)
0927 18:42:42.581633 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.581638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.594080 (+ 12442us) server_negotiation.cc:407] Connection header received
0927 18:42:42.594779 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.594782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.594833 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.594881 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.596970 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.597731 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.598953 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.599315 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.599387 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.600988 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.600992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.600995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.601018 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.602548 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.603199 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.603202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.603204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.603297 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.603740 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.603744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.603745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.603958 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.604081 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.604387 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.604489 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:42:42.607540 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:42.592337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.18.235.188:46221)
0927 18:42:42.592439 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:42.592443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:42.598320 (+ 5877us) server_negotiation.cc:407] Connection header received
0927 18:42:42.598507 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:42.598510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:42.598556 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:42.598603 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:42.600162 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.600890 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.602041 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:42.602356 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:42.602426 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:42.604367 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:42.604370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:42.604373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:42.604396 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:42.606260 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.606610 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.606613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.606616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.606692 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:42.606884 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:42.606886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:42.606888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:42.607065 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:42.607177 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:42.607302 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 18:42:42.607404 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:42:43.570257 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:43.569830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:57815)
0927 18:42:43.570096 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:43.570163 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":70,"threads_started":1}
I20260927 18:42:43.570370 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:43.569972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:41667)
0927 18:42:43.570167 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:43.570235 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
W20260927 18:42:43.570616 26583 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:39097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
W20260927 18:42:43.570854 26719 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:39097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:43.572324 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:43.571897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:39993)
0927 18:42:43.572158 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:43.572239 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":60,"threads_started":1}
W20260927 18:42:43.572554 26446 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:39097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:44.079533 26789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c5d48128c364721a4a3d3a537a3dc20)
I20260927 18:42:44.079772 26789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } }
I20260927 18:42:44.080695 26789 leader_election.cc:288] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097), 141243b2f105401f95d3938889172241 (127.18.235.189:44241)
I20260927 18:42:44.081331 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.080861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40118)
0927 18:42:44.081160 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.081245 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":57,"threads_started":1}
W20260927 18:42:44.081488 26250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:44.082260 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.082047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40130)
0927 18:42:44.082138 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.082189 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:42:44.082468 26250 leader_election.cc:334] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097): Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:44.082505 26794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c5d48128c364721a4a3d3a537a3dc20)
I20260927 18:42:44.082644 26794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } }
I20260927 18:42:44.083627 26794 leader_election.cc:288] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097), 393d86ba2f354a2b9dfc8e3d45e0714d (127.18.235.188:46221)
I20260927 18:42:44.084045 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.083701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40132)
0927 18:42:44.083891 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.083960 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":55,"threads_started":1}
W20260927 18:42:44.084381 26068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:44.085170 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.084938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40144)
0927 18:42:44.085030 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.085078 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:42:44.085191 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.081004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.18.235.189:44241)
0927 18:42:44.081323 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.081327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.081343 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:42:44.081397 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.081400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.081443 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.081523 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.081822 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.082529 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.084032 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.084613 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.084765 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.084815 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.084973 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.085041 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":251,"thread_start_us":124,"threads_started":1}
I20260927 18:42:44.085191 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.080916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:44241 (local address 127.0.0.1:33960)
0927 18:42:44.081194 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.081208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.081320 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.081557 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.081560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.081584 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.081690 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.081698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.082688 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.082691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.083895 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.083907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.084761 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.084765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.084865 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.084885 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.084948 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.085019 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
W20260927 18:42:44.085284 26068 leader_election.cc:334] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097): Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:44.085690 26094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "141243b2f105401f95d3938889172241" is_pre_election: true
I20260927 18:42:44.085852 26094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 393d86ba2f354a2b9dfc8e3d45e0714d in term 1.
I20260927 18:42:44.086154 26253 leader_election.cc:302] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 141243b2f105401f95d3938889172241, 393d86ba2f354a2b9dfc8e3d45e0714d; no voters: 4c5d48128c364721a4a3d3a537a3dc20
I20260927 18:42:44.086337 26789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:42:44.086392 26789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c5d48128c364721a4a3d3a537a3dc20)
I20260927 18:42:44.086434 26789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:42:44.087477 26789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } }
I20260927 18:42:44.087843 26789 leader_election.cc:288] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [CANDIDATE]: Term 2 election: Requested vote from peers 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097), 141243b2f105401f95d3938889172241 (127.18.235.189:44241)
I20260927 18:42:44.087970 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.084024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.18.235.188:46221)
0927 18:42:44.084250 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.084256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.084266 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:42:44.084303 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.084305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.084344 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.084432 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.084741 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.085766 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.086941 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.087354 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.087443 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.087475 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.087693 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.087753 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260927 18:42:44.088188 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.087926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40154)
0927 18:42:44.088076 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.088146 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 18:42:44.088143 26094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "141243b2f105401f95d3938889172241"
I20260927 18:42:44.088253 26094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:42:44.088757 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.088563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40170)
0927 18:42:44.088652 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.088696 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:42:44.088852 26250 leader_election.cc:334] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097): Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
I20260927 18:42:44.089171 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.083788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:46221 (local address 127.0.0.1:40112)
0927 18:42:44.084009 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.084021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.084112 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.084458 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.084461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.084524 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.084611 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.084616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.085879 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.085882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.086808 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.086818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.087472 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.087476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.087566 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.087590 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.087680 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.087749 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":49,"threads_started":1}
I20260927 18:42:44.089413 26094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 393d86ba2f354a2b9dfc8e3d45e0714d in term 2.
I20260927 18:42:44.089570 26270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "141243b2f105401f95d3938889172241" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" is_pre_election: true
I20260927 18:42:44.089643 26253 leader_election.cc:302] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 141243b2f105401f95d3938889172241, 393d86ba2f354a2b9dfc8e3d45e0714d; no voters: 4c5d48128c364721a4a3d3a537a3dc20
I20260927 18:42:44.089767 26270 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 141243b2f105401f95d3938889172241 in current term 2: Already voted for candidate 393d86ba2f354a2b9dfc8e3d45e0714d in this term.
I20260927 18:42:44.089857 26789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:42:44.090046 26789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [term 2 LEADER]: Becoming Leader. State: Replica: 393d86ba2f354a2b9dfc8e3d45e0714d, State: Running, Role: LEADER
I20260927 18:42:44.090112 26069 leader_election.cc:302] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 141243b2f105401f95d3938889172241; no voters: 393d86ba2f354a2b9dfc8e3d45e0714d, 4c5d48128c364721a4a3d3a537a3dc20
I20260927 18:42:44.090301 26794 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:42:44.090287 26789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } }
I20260927 18:42:44.091553 26799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 393d86ba2f354a2b9dfc8e3d45e0714d. Latest consensus state: current_term: 2 leader_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:44.091686 26799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:44.091946 26801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:44.094244 26801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:44.094636 26801 catalog_manager.cc:1259] Loaded cluster ID: 4735f9a81ed74473a2fdc18710c295db
I20260927 18:42:44.094679 26801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:44.095127 26801 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:44.095481 26801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Loaded TSK: 0
I20260927 18:42:44.096040 26801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:42:44.153802 26265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:44.156757 26265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:44.159533 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.159301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40192)
0927 18:42:44.159422 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.159472 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:42:44.159571 26094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [term 2 FOLLOWER]: Refusing update from remote peer 393d86ba2f354a2b9dfc8e3d45e0714d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 18:42:44.160032 26789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [LEADER]: Connected to new peer: Peer: permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 18:42:44.160138 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.159928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.0.0.1:40200)
0927 18:42:44.160040 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.160085 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:42:44.160250 26250 consensus_peers.cc:487] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d -> Peer 4c5d48128c364721a4a3d3a537a3dc20 (127.18.235.190:39097): Couldn't send request to peer 4c5d48128c364721a4a3d3a537a3dc20. Status: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:42:44.161352 26794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 393d86ba2f354a2b9dfc8e3d45e0714d. Latest consensus state: current_term: 2 leader_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:44.161460 26794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:44.162369 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:44.162483 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:44.162662 26794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4c5d48128c364721a4a3d3a537a3dc20" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39097 } } peers { permanent_uuid: "141243b2f105401f95d3938889172241" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 44241 } attrs { promote: false } } peers { permanent_uuid: "393d86ba2f354a2b9dfc8e3d45e0714d" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 46221 } attrs { promote: false } } }
I20260927 18:42:44.162768 26794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141243b2f105401f95d3938889172241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:42:44.171918 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.167974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:34875 (local address 127.0.0.1:44532)
0927 18:42:44.168074 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.168089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.168160 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.168876 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.168879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.168894 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.168988 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.168993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.169917 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.169920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.170657 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.170665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.171567 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.171570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.171657 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.171682 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.171725 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.171797 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260927 18:42:44.172397 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.168187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:34049 (local address 127.0.0.1:52784)
0927 18:42:44.168398 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.168410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.168478 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.169297 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.169300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.169318 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.169393 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.169399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.170363 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.170366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.171343 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.171352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.172069 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.172073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.172160 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.172183 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.172224 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.172285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 18:42:44.172420 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.168109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.18.235.131:34875)
0927 18:42:44.168629 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.168633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.168647 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:42:44.168694 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.168697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.168763 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.169077 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.169199 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.169824 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.170793 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.171441 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.172180 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.172234 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.172263 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.172313 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":81,"threads_started":1}
I20260927 18:42:44.172829 26652 tablet_service.cc:1467] Processing CreateTablet for tablet 06cf3e26600246fb9aa2c9917d823caa (DEFAULT_TABLE table=test-table [id=8eb616a675f74381a8243166b3f6547c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:44.172914 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.168624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.18.235.129:34049)
0927 18:42:44.169051 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.169056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.169067 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:44.169135 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.169137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.169184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.169264 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.169494 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.170233 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.171482 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.171947 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.172643 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.172686 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.172717 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.172780 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":60,"threads_started":1}
I20260927 18:42:44.173071 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.168473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:35211 (local address 127.0.0.1:47910)
0927 18:42:44.169244 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.169257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.169331 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.169686 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.169689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.169705 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.169783 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.169788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.170731 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.170735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.171806 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.171813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.172723 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.172726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.172805 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.172825 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.172897 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.172962 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":83,"threads_started":1}
I20260927 18:42:44.173098 26652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cf3e26600246fb9aa2c9917d823caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:44.173346 26379 tablet_service.cc:1467] Processing CreateTablet for tablet 06cf3e26600246fb9aa2c9917d823caa (DEFAULT_TABLE table=test-table [id=8eb616a675f74381a8243166b3f6547c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:44.173558 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.168808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.18.235.130:35211)
0927 18:42:44.169016 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.169020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.169415 (+ 395us) server_negotiation.cc:407] Connection header received
0927 18:42:44.169467 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.169470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.169517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.169588 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.169893 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.170620 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.171918 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.172514 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.172623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.172668 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.173302 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.173382 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260927 18:42:44.173571 26379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cf3e26600246fb9aa2c9917d823caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:44.175412 26812 tablet_bootstrap.cc:492] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Bootstrap starting.
I20260927 18:42:44.175678 26813 tablet_bootstrap.cc:492] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1: Bootstrap starting.
I20260927 18:42:44.176007 26516 tablet_service.cc:1467] Processing CreateTablet for tablet 06cf3e26600246fb9aa2c9917d823caa (DEFAULT_TABLE table=test-table [id=8eb616a675f74381a8243166b3f6547c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:44.176254 26516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06cf3e26600246fb9aa2c9917d823caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:44.176260 26812 tablet_bootstrap.cc:654] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:44.176467 26813 tablet_bootstrap.cc:654] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:44.177347 26812 tablet_bootstrap.cc:492] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: No bootstrap required, opened a new log
I20260927 18:42:44.177415 26812 ts_tablet_manager.cc:1397] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:44.177664 26813 tablet_bootstrap.cc:492] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1: No bootstrap required, opened a new log
I20260927 18:42:44.177743 26812 raft_consensus.cc:348] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.177829 26812 raft_consensus.cc:374] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:44.177852 26812 raft_consensus.cc:729] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43588c46c044d5c9a72e29ce096646f, State: Initialized, Role: FOLLOWER
I20260927 18:42:44.177798 26813 ts_tablet_manager.cc:1397] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:44.177946 26812 consensus_queue.cc:260] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.178249 26812 ts_tablet_manager.cc:1428] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:44.178706 26813 raft_consensus.cc:348] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.178886 26813 raft_consensus.cc:374] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:44.178963 26813 raft_consensus.cc:729] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64db06eea6db4d66b6267908472f90f1, State: Initialized, Role: FOLLOWER
I20260927 18:42:44.179160 26813 consensus_queue.cc:260] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.179479 26813 ts_tablet_manager.cc:1428] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:44.180545 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.179808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:38369)
0927 18:42:44.180385 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.180444 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":90,"threads_started":1}
I20260927 18:42:44.180585 26718 heartbeater.cc:503] Master 127.18.235.188:46221 was elected leader, sending a full tablet report...
I20260927 18:42:44.180852 26445 heartbeater.cc:503] Master 127.18.235.188:46221 was elected leader, sending a full tablet report...
I20260927 18:42:44.180883 26817 tablet_bootstrap.cc:492] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Bootstrap starting.
I20260927 18:42:44.181790 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.179849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:38133)
0927 18:42:44.181617 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.181690 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1709,"thread_start_us":111,"threads_started":1}
I20260927 18:42:44.182535 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.181787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:43349)
0927 18:42:44.182368 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.182419 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511}
I20260927 18:42:44.182595 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.182181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:43413)
0927 18:42:44.182467 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.182522 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
I20260927 18:42:44.183100 26817 tablet_bootstrap.cc:654] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:44.184115 26817 tablet_bootstrap.cc:492] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: No bootstrap required, opened a new log
I20260927 18:42:44.184183 26817 ts_tablet_manager.cc:1397] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:42:44.184473 26817 raft_consensus.cc:348] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.184553 26817 raft_consensus.cc:374] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:44.184578 26817 raft_consensus.cc:729] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632340f3f2049d7ad8beee6e97ba050, State: Initialized, Role: FOLLOWER
I20260927 18:42:44.184665 26817 consensus_queue.cc:260] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.184897 26817 ts_tablet_manager.cc:1428] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:44.185536 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.185195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:57537)
0927 18:42:44.185398 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.185451 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260927 18:42:44.186303 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.186073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:40691)
0927 18:42:44.186189 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.186240 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 18:42:44.186362 26582 heartbeater.cc:503] Master 127.18.235.188:46221 was elected leader, sending a full tablet report...
W20260927 18:42:44.321235 26721 tablet.cc:2369] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:44.325613 26816 raft_consensus.cc:482] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:44.325713 26816 raft_consensus.cc:504] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.326747 26816 leader_election.cc:288] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049), e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211)
I20260927 18:42:44.330401 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.327089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:39301 (local address 127.18.235.130:35211)
0927 18:42:44.327214 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.327219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.327241 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:44.327292 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.327295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.327338 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.327408 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.327750 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.328338 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.329490 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.329879 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.329968 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.330018 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.330203 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.330287 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:42:44.330397 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.326920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:35211 (local address 127.18.235.131:39301)
0927 18:42:44.327112 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.327128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.327214 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.327454 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.327457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.327481 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.327616 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.327623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.328431 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.328435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.329376 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.329387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.329990 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.329994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.330091 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.330115 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.330172 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.330234 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":46}
I20260927 18:42:44.330423 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.327028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:45613 (local address 127.18.235.129:34049)
0927 18:42:44.327140 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.327144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.327399 (+ 255us) server_negotiation.cc:407] Connection header received
0927 18:42:44.327514 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.327517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.327562 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.327645 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:44.327928 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.328452 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.329530 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.329923 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.330031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.330078 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.330251 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.330325 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:42:44.330416 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.326920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:34049 (local address 127.18.235.131:45613)
0927 18:42:44.327324 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.327336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:44.327415 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:44.327668 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:44.327671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:44.327687 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:44.327801 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.327808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.328558 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.328560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.329410 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:44.329420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.330055 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.330059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:44.330136 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.330152 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:42:44.330224 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:42:44.330285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 18:42:44.330842 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf3e26600246fb9aa2c9917d823caa" candidate_uuid: "a43588c46c044d5c9a72e29ce096646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64db06eea6db4d66b6267908472f90f1" is_pre_election: true
I20260927 18:42:44.330986 26399 raft_consensus.cc:2454] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43588c46c044d5c9a72e29ce096646f in term 0.
I20260927 18:42:44.330983 26536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf3e26600246fb9aa2c9917d823caa" candidate_uuid: "a43588c46c044d5c9a72e29ce096646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632340f3f2049d7ad8beee6e97ba050" is_pre_election: true
I20260927 18:42:44.331219 26536 raft_consensus.cc:2454] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43588c46c044d5c9a72e29ce096646f in term 0.
I20260927 18:42:44.331578 26605 leader_election.cc:302] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a43588c46c044d5c9a72e29ce096646f, e632340f3f2049d7ad8beee6e97ba050; no voters:
I20260927 18:42:44.331792 26816 raft_consensus.cc:2781] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:44.331844 26816 raft_consensus.cc:482] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:44.331871 26816 raft_consensus.cc:3037] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:44.332798 26816 raft_consensus.cc:504] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.333084 26816 leader_election.cc:288] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [CANDIDATE]: Term 1 election: Requested vote from peers 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049), e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211)
I20260927 18:42:44.333416 26536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf3e26600246fb9aa2c9917d823caa" candidate_uuid: "a43588c46c044d5c9a72e29ce096646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632340f3f2049d7ad8beee6e97ba050"
I20260927 18:42:44.333416 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06cf3e26600246fb9aa2c9917d823caa" candidate_uuid: "a43588c46c044d5c9a72e29ce096646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64db06eea6db4d66b6267908472f90f1"
I20260927 18:42:44.333525 26399 raft_consensus.cc:3037] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:44.333530 26536 raft_consensus.cc:3037] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:44.334559 26536 raft_consensus.cc:2454] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43588c46c044d5c9a72e29ce096646f in term 1.
I20260927 18:42:44.334559 26399 raft_consensus.cc:2454] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43588c46c044d5c9a72e29ce096646f in term 1.
I20260927 18:42:44.334820 26605 leader_election.cc:302] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a43588c46c044d5c9a72e29ce096646f, e632340f3f2049d7ad8beee6e97ba050; no voters:
I20260927 18:42:44.334985 26816 raft_consensus.cc:2781] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:44.335187 26816 raft_consensus.cc:686] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 1 LEADER]: Becoming Leader. State: Replica: a43588c46c044d5c9a72e29ce096646f, State: Running, Role: LEADER
I20260927 18:42:44.335417 26816 consensus_queue.cc:237] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } }
I20260927 18:42:44.335740 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.335564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:46939)
0927 18:42:44.335641 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.335690 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 18:42:44.337484 26265 catalog_manager.cc:5495] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f reported cstate change: term changed from 0 to 1, leader changed from <none> to a43588c46c044d5c9a72e29ce096646f (127.18.235.131). New cstate: current_term: 1 leader_uuid: "a43588c46c044d5c9a72e29ce096646f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43588c46c044d5c9a72e29ce096646f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 34875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:44.356899 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.353938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.18.235.131:34875)
0927 18:42:44.354039 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:44.354042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:44.354055 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:44.354175 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:44.354177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:44.354225 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:44.354303 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:44.354468 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.354941 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.355680 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:44.355924 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:44.355995 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:44.356166 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:44.356580 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:44.356667 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:44.356725 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 18:42:44.356767 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:42:44.359253 26399 raft_consensus.cc:1261] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Refusing update from remote peer a43588c46c044d5c9a72e29ce096646f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:44.359562 26536 raft_consensus.cc:1261] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Refusing update from remote peer a43588c46c044d5c9a72e29ce096646f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:44.359691 26825 consensus_queue.cc:1035] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64db06eea6db4d66b6267908472f90f1" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:44.360021 26816 consensus_queue.cc:1035] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632340f3f2049d7ad8beee6e97ba050" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:44.361268 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.360965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:38335)
0927 18:42:44.361128 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.361188 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 18:42:44.361840 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.361595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:48135)
0927 18:42:44.361730 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.361777 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:42:44.365288 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.365027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.130:58401)
0927 18:42:44.365179 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.365227 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 18:42:44.366580 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.366372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.131:54561)
0927 18:42:44.366470 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.366520 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 18:42:44.366771 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:44.366540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:39097 (local address 127.18.235.129:38337)
0927 18:42:44.366668 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.366713 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:39097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 18:42:44.372121 26264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:44.372246 26264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:44.375604 26264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 393d86ba2f354a2b9dfc8e3d45e0714d: Sending DeleteTablet for 3 replicas of tablet 06cf3e26600246fb9aa2c9917d823caa
I20260927 18:42:44.376216 26379 tablet_service.cc:1514] Processing DeleteTablet for tablet 06cf3e26600246fb9aa2c9917d823caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:44 UTC) from {username='oryx', principal='oryx/127.18.235.188@KRBTEST.COM'} at 127.0.0.1:52784
I20260927 18:42:44.376217 26516 tablet_service.cc:1514] Processing DeleteTablet for tablet 06cf3e26600246fb9aa2c9917d823caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:44 UTC) from {username='oryx', principal='oryx/127.18.235.188@KRBTEST.COM'} at 127.0.0.1:47910
I20260927 18:42:44.376490 26838 tablet_replica.cc:330] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: stopping tablet replica
I20260927 18:42:44.376606 26838 raft_consensus.cc:2229] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:44.376618 26652 tablet_service.cc:1514] Processing DeleteTablet for tablet 06cf3e26600246fb9aa2c9917d823caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:44 UTC) from {username='oryx', principal='oryx/127.18.235.188@KRBTEST.COM'} at 127.0.0.1:44532
I20260927 18:42:44.376734 26838 raft_consensus.cc:2258] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:44.376849 26839 tablet_replica.cc:330] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1: stopping tablet replica
I20260927 18:42:44.376993 26839 raft_consensus.cc:2229] T 06cf3e26600246fb9aa2c9917d823caa P 64db06eea6db4d66b6267908472f90f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:44.377246 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 26228
I20260927 18:42:44.378784 26838 ts_tablet_manager.cc:1905] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:44.381206 26838 ts_tablet_manager.cc:1918] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:44.381279 26838 log.cc:1196] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/06cf3e26600246fb9aa2c9917d823caa
I20260927 18:42:44.381522 26838 ts_tablet_manager.cc:1939] T 06cf3e26600246fb9aa2c9917d823caa P e632340f3f2049d7ad8beee6e97ba050: Deleting consensus metadata
I20260927 18:42:44.381673 26840 tablet_replica.cc:330] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: stopping tablet replica
I20260927 18:42:44.381817 26840 raft_consensus.cc:2229] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:44.382108 26840 raft_consensus.cc:2258] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:44.382090 26250 catalog_manager.cc:4841] TS e632340f3f2049d7ad8beee6e97ba050 (127.18.235.130:35211): tablet 06cf3e26600246fb9aa2c9917d823caa (table test-table [id=8eb616a675f74381a8243166b3f6547c]) successfully deleted
I20260927 18:42:44.382750 26840 ts_tablet_manager.cc:1905] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:42:44.384143 26809 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:44.383878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:34049 (local address 127.0.0.1:52798)
0927 18:42:44.383991 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:44.384052 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:34049: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 18:42:44.384155 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 26453
I20260927 18:42:44.385174 26840 ts_tablet_manager.cc:1918] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:44.385245 26840 log.cc:1196] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/06cf3e26600246fb9aa2c9917d823caa
W20260927 18:42:44.385439 26253 catalog_manager.cc:4584] TS 64db06eea6db4d66b6267908472f90f1 (127.18.235.129:34049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06cf3e26600246fb9aa2c9917d823caa: Network error: Client connection negotiation failed: client connection to 127.18.235.129:34049: connect: Connection reset by peer (error 104)
I20260927 18:42:44.385512 26840 ts_tablet_manager.cc:1939] T 06cf3e26600246fb9aa2c9917d823caa P a43588c46c044d5c9a72e29ce096646f: Deleting consensus metadata
I20260927 18:42:44.386232 26253 catalog_manager.cc:4841] TS a43588c46c044d5c9a72e29ce096646f (127.18.235.131:34875): tablet 06cf3e26600246fb9aa2c9917d823caa (table test-table [id=8eb616a675f74381a8243166b3f6547c]) successfully deleted
I20260927 18:42:44.391496 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 26589
I20260927 18:42:44.397332 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 25947
I20260927 18:42:44.401713 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 26055
2026-09-27T18:42:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5086 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:44 dist-test-slave-6lr6 krb5kdc[26843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:44 dist-test-slave-6lr6 krb5kdc[26843](info): set up 2 sockets
Sep 27 18:42:44 dist-test-slave-6lr6 krb5kdc[26843](info): commencing operation
krb5kdc: starting...
W20260927 18:42:45.345253 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:45 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab.
Entry for principal oryx/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab.
I20260927 18:42:45.449852 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:41329
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab
--principal=oryx/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:45.551017 26856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:45.551229 26856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:45.551263 26856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:45.555173 26856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:45.555217 26856 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:42:45.555233 26856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:45.555248 26856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:45.555269 26856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:45.555285 26856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:45.560128 26856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:41329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.190.keytab
--principal=oryx/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:45.560920 26856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:45.561741 26856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:45.568958 26862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:45.569011 26864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:45.569141 26856 server_base.cc:1034] running on GCE node
W20260927 18:42:45.568959 26861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:45.569691 26856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:45.570606 26856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:45.571759 26856 hybrid_clock.cc:648] HybridClock initialized: now 1790534565571748 us; error 23 us; skew 500 ppm
Sep 27 18:42:45 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534565, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:45.574156 26856 init.cc:377] Logged in from keytab as oryx/127.18.235.190@KRBTEST.COM (short username oryx)
I20260927 18:42:45.574878 26856 webserver.cc:460] Webserver started at http://127.18.235.190:40601/ using document root <none> and password file <none>
I20260927 18:42:45.575340 26856 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:45.575408 26856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:45.575585 26856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:45.577378 26856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "51e9cfed80d5429c860812f5113f3f40"
format_stamp: "Formatted at 2026-09-27 18:42:45 on dist-test-slave-6lr6"
server_key: "cd245ba0d7b098bf1da6676743759645"
server_key_iv: "fba01ebf5559ec8fc79fa7161dfbcc6b"
server_key_version: "encryptionkey@0"
I20260927 18:42:45.577849 26856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "51e9cfed80d5429c860812f5113f3f40"
format_stamp: "Formatted at 2026-09-27 18:42:45 on dist-test-slave-6lr6"
server_key: "cd245ba0d7b098bf1da6676743759645"
server_key_iv: "fba01ebf5559ec8fc79fa7161dfbcc6b"
server_key_version: "encryptionkey@0"
I20260927 18:42:45.580945 26856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:42:45.582960 26871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:45.583978 26856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:45.584079 26856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "51e9cfed80d5429c860812f5113f3f40"
format_stamp: "Formatted at 2026-09-27 18:42:45 on dist-test-slave-6lr6"
server_key: "cd245ba0d7b098bf1da6676743759645"
server_key_iv: "fba01ebf5559ec8fc79fa7161dfbcc6b"
server_key_version: "encryptionkey@0"
I20260927 18:42:45.584156 26856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:45.601579 26856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:45.603788 26856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:45.603930 26856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:45.611371 26856 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:41329
I20260927 18:42:45.611395 26933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:41329 every 8 connection(s)
I20260927 18:42:45.612131 26856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:45.614262 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 26856
I20260927 18:42:45.614398 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:45.614672 19374 external_mini_cluster.cc:1428] Setting key e70e718afd9ab295378c4d4d695fbc6f
I20260927 18:42:45.614722 26934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:45.620296 26934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Bootstrap starting.
I20260927 18:42:45.622411 26934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Neither blocks nor log segments found. Creating new log.
Sep 27 18:42:45 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:45.623184 26934 log.cc:824] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:45.625260 26934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: No bootstrap required, opened a new log
I20260927 18:42:45.628271 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:45.615931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.18.235.190:41329)
0927 18:42:45.616317 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:45.616330 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:45.616360 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:42:45.617084 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:45.617121 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:45.617419 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:45.617692 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:45.618424 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.619358 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.620143 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.620605 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.620758 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:45.623081 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:45.623085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:45.623097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:45.623121 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:45.625475 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:45.625881 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:45.625888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:45.625893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:45.626044 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:45.626280 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:45.626284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:45.626286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:45.626749 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:45.626935 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:45.627120 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 18:42:45.627403 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260927 18:42:45.628999 26934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e9cfed80d5429c860812f5113f3f40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41329 } }
I20260927 18:42:45.629273 26934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:45.629338 26934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e9cfed80d5429c860812f5113f3f40, State: Initialized, Role: FOLLOWER
I20260927 18:42:45.629743 26934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e9cfed80d5429c860812f5113f3f40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41329 } }
I20260927 18:42:45.629859 26934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:45.629899 26934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:45.629962 26934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:45.630823 26934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e9cfed80d5429c860812f5113f3f40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41329 } }
I20260927 18:42:45.631094 26934 leader_election.cc:302] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e9cfed80d5429c860812f5113f3f40; no voters:
I20260927 18:42:45.631328 26934 leader_election.cc:288] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:45.631414 26939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:45.631593 26939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [term 1 LEADER]: Becoming Leader. State: Replica: 51e9cfed80d5429c860812f5113f3f40, State: Running, Role: LEADER
I20260927 18:42:45.631922 26939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e9cfed80d5429c860812f5113f3f40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41329 } }
I20260927 18:42:45.632453 26934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:45.633589 26940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e9cfed80d5429c860812f5113f3f40. Latest consensus state: current_term: 1 leader_uuid: "51e9cfed80d5429c860812f5113f3f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e9cfed80d5429c860812f5113f3f40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41329 } } }
I20260927 18:42:45.633730 26940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:45.634451 26948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:45.633602 26941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e9cfed80d5429c860812f5113f3f40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e9cfed80d5429c860812f5113f3f40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41329 } } }
I20260927 18:42:45.634958 26941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:45.637279 26948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:45.642447 26948 catalog_manager.cc:1347] Generated new cluster ID: 996145dc43f34602acd4b046ac0c8f7a
I20260927 18:42:45.642544 26948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:45.654565 26948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:45.655705 26948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:45.664124 26948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Generated new TSK 0
I20260927 18:42:45.664726 26948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab.
Entry for principal oryx/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab.
I20260927 18:42:45.701917 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41329
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:45.832657 26960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:45.832875 26960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:45.832914 26960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:45.836799 26960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:45.836846 26960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:45.836882 26960 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:45.836917 26960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:45.841663 26960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:41329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:45.842442 26960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:45.843257 26960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:45.851409 26965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:45.851402 26966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:45.851402 26968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:45.851603 26960 server_base.cc:1034] running on GCE node
I20260927 18:42:45.852032 26960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:45.852530 26960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:45.853677 26960 hybrid_clock.cc:648] HybridClock initialized: now 1790534565853661 us; error 26 us; skew 500 ppm
Sep 27 18:42:45 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534565, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:45.859225 26960 init.cc:377] Logged in from keytab as oryx/127.18.235.129@KRBTEST.COM (short username oryx)
I20260927 18:42:45.859956 26960 webserver.cc:460] Webserver started at http://127.18.235.129:35555/ using document root <none> and password file <none>
I20260927 18:42:45.860424 26960 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:45.860484 26960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:45.860661 26960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:45.862489 26960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
format_stamp: "Formatted at 2026-09-27 18:42:45 on dist-test-slave-6lr6"
server_key: "7831296e663e830e389c74d86b38694b"
server_key_iv: "67ffbccc80e05ab31f474ede19d7f051"
server_key_version: "encryptionkey@0"
I20260927 18:42:45.862986 26960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
format_stamp: "Formatted at 2026-09-27 18:42:45 on dist-test-slave-6lr6"
server_key: "7831296e663e830e389c74d86b38694b"
server_key_iv: "67ffbccc80e05ab31f474ede19d7f051"
server_key_version: "encryptionkey@0"
I20260927 18:42:45.866410 26960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:45.868566 26975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:45.869632 26960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:45.869758 26960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
format_stamp: "Formatted at 2026-09-27 18:42:45 on dist-test-slave-6lr6"
server_key: "7831296e663e830e389c74d86b38694b"
server_key_iv: "67ffbccc80e05ab31f474ede19d7f051"
server_key_version: "encryptionkey@0"
I20260927 18:42:45.869848 26960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:45.903270 26960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:45.905790 26960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:45.905956 26960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:45.907246 26960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:45.907298 26960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:45.907351 26960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:45.907383 26960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:45 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534565, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:45.918001 26960 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:43835
I20260927 18:42:45.918098 27090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:43835 every 8 connection(s)
I20260927 18:42:45.918937 26960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:45.920531 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:45.907935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:40115 (local address 127.18.235.190:41329)
0927 18:42:45.908043 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:45.908047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:45.909036 (+ 989us) server_negotiation.cc:407] Connection header received
0927 18:42:45.909898 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:45.909901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:45.909958 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:45.910026 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:45.911357 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.912151 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.913782 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.914133 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.914235 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:45.916975 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:45.916978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:45.916982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:45.917013 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:45.918443 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:45.918967 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:45.918969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:45.918971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:45.919052 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:45.919439 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:45.919443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:45.919445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:45.919676 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:45.919776 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:45.920238 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 18:42:45.920362 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:42:45.921208 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:45.908456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.18.235.129:40115)
0927 18:42:45.908900 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:45.908938 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:45.909704 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:45.910202 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:45.910211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:45.910654 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:45.911225 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:45.911240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.912291 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.912295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:45.913658 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:45.913671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.914282 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.914287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:45.914483 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:45.915035 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:45.915056 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:45.916809 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:45.918541 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:45.918547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:45.918559 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:45.918865 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:45.919123 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:45.919127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:45.919129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:45.919305 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:45.919785 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:45.919791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:45.920008 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 18:42:45.920172 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 18:42:45.920444 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260927 18:42:45.922286 27091 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41329
I20260927 18:42:45.922413 27091 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:45.922900 27091 heartbeater.cc:511] Master 127.18.235.190:41329 requested a full tablet report, sending...
I20260927 18:42:45.924350 26887 ts_manager.cc:194] Registered new tserver with Master: a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835)
I20260927 18:42:45.925532 26887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:40115
I20260927 18:42:45.927192 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 26960
I20260927 18:42:45.927320 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:45.927660 19374 external_mini_cluster.cc:1428] Setting key 521b03444c14a92412b65ef241124361
I20260927 18:42:45.930826 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:45.924305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:33117 (local address 127.18.235.190:41329)
0927 18:42:45.924406 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:45.924410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:45.924602 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:42:45.924699 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:45.924702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:45.924740 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:45.924796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:45.925179 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.925683 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.926576 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.926797 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.926883 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:45.927953 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:45.927957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:45.927959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:45.927985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:45.929452 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:45.929800 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:45.929804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:45.929805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:45.929892 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:45.930099 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:45.930102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:45.930104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:45.930304 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:45.930404 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:45.930610 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:42:45.930723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:42:45.930850 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:45.924258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.18.235.129:33117)
0927 18:42:45.924522 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:45.924535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:45.924622 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:45.924818 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:45.924821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:45.924984 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:45.925066 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:45.925071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.925792 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.925796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:45.926477 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:45.926487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:45.926904 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:45.926908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:45.926994 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:45.927461 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:45.927474 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:45.927810 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:45.929547 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:45.929549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:45.929551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:45.929718 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:45.929968 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:45.929970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:45.929971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:45.930031 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:45.930383 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:45.930385 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:45.930449 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 18:42:45.930547 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:42:45.930670 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab.
Entry for principal oryx/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab.
I20260927 18:42:45.949530 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41329
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:46.051493 27100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:46.051726 27100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:46.051790 27100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:46.055732 27100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:46.055788 27100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:46.055848 27100 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:46.055907 27100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:46.060767 27100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:41329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:46.061662 27100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:46.062523 27100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:46.070925 27105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:46.070953 27106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:46.071128 27100 server_base.cc:1034] running on GCE node
W20260927 18:42:46.070996 27108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:46.071632 27100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:46.072191 27100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:46.073339 27100 hybrid_clock.cc:648] HybridClock initialized: now 1790534566073326 us; error 22 us; skew 500 ppm
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:46.075831 27100 init.cc:377] Logged in from keytab as oryx/127.18.235.130@KRBTEST.COM (short username oryx)
I20260927 18:42:46.076557 27100 webserver.cc:460] Webserver started at http://127.18.235.130:35209/ using document root <none> and password file <none>
I20260927 18:42:46.077001 27100 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:46.077051 27100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:46.077235 27100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:46.078977 27100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "e6dfc6f35ca54a8bab6c092dd1859678"
format_stamp: "Formatted at 2026-09-27 18:42:46 on dist-test-slave-6lr6"
server_key: "533867dbaa9064ce47a9969a82842621"
server_key_iv: "2c2f886a94d621b62a1a355900487b3e"
server_key_version: "encryptionkey@0"
I20260927 18:42:46.079430 27100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "e6dfc6f35ca54a8bab6c092dd1859678"
format_stamp: "Formatted at 2026-09-27 18:42:46 on dist-test-slave-6lr6"
server_key: "533867dbaa9064ce47a9969a82842621"
server_key_iv: "2c2f886a94d621b62a1a355900487b3e"
server_key_version: "encryptionkey@0"
I20260927 18:42:46.082571 27100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:46.084610 27115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:46.085721 27100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:46.085814 27100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "e6dfc6f35ca54a8bab6c092dd1859678"
format_stamp: "Formatted at 2026-09-27 18:42:46 on dist-test-slave-6lr6"
server_key: "533867dbaa9064ce47a9969a82842621"
server_key_iv: "2c2f886a94d621b62a1a355900487b3e"
server_key_version: "encryptionkey@0"
I20260927 18:42:46.085904 27100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:46.122576 27100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:46.125066 27100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:46.125234 27100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:46.126464 27100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:46.126510 27100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:46.126559 27100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:46.126587 27100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:46.135735 27100 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:39105
I20260927 18:42:46.135772 27230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:39105 every 8 connection(s)
I20260927 18:42:46.136602 27100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:46.138183 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.127135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:50521 (local address 127.18.235.190:41329)
0927 18:42:46.127289 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.127294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.128106 (+ 812us) server_negotiation.cc:407] Connection header received
0927 18:42:46.129087 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.129091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.129170 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.129261 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.130412 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.130920 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.131974 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.132202 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.132290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.134914 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.134918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.134921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.134947 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.136278 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.136804 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.136806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.136808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.136882 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.137282 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.137286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.137287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.137516 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.137596 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.137972 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.138063 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:42:46.138813 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.127536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.18.235.130:50521)
0927 18:42:46.127959 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.128004 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.128864 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.129494 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.129502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.129859 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:46.130281 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.130293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.131028 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.131032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.131865 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.131874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.132357 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.132361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.132499 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.132986 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:46.133002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:46.134740 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:46.136391 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.136396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.136408 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.136710 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.136978 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.136981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.136983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.137170 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.137607 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:46.137611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:46.137800 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.137939 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.138141 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260927 18:42:46.139881 27231 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41329
I20260927 18:42:46.140003 27231 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:46.140439 27231 heartbeater.cc:511] Master 127.18.235.190:41329 requested a full tablet report, sending...
I20260927 18:42:46.141320 26888 ts_manager.cc:194] Registered new tserver with Master: e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105)
I20260927 18:42:46.141930 26888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:50521
I20260927 18:42:46.143985 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 27100
I20260927 18:42:46.144078 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:46.144362 19374 external_mini_cluster.cc:1428] Setting key 79124df180ba4ee46d83bcb0a8ae0c0b
I20260927 18:42:46.148175 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.141625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:46351 (local address 127.18.235.190:41329)
0927 18:42:46.141713 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.141716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.142323 (+ 607us) server_negotiation.cc:407] Connection header received
0927 18:42:46.142391 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.142394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.142433 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.142491 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.142926 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.143392 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.144184 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.144477 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.144568 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.145561 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.145563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.145565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.145586 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.146846 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.147189 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.147193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.147194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.147273 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.147530 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.147532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.147533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.147698 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.147780 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.147999 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.148079 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:42:46.148257 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.141567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.18.235.130:46351)
0927 18:42:46.142254 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.142265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.142334 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.142523 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.142526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.142715 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:46.142823 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.142830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.143503 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.143506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.144081 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.144092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.144576 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.144578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.144644 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.145061 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:46.145073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:46.145452 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:46.146935 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.146938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.146939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.147105 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.147356 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.147359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.147361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.147454 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.147783 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:46.147785 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:46.147868 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.147963 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.148082 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for oryx/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab.
Entry for principal oryx/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab.
I20260927 18:42:46.168195 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41329
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:46.269667 27240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:46.269874 27240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:46.269922 27240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:46.273751 27240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:46.273806 27240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:46.273874 27240 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:42:46.273922 27240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:46.278759 27240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.235.190:41329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:46.279556 27240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:46.280400 27240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:46.288663 27248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:46.288650 27245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:46.288635 27246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:46.289592 27240 server_base.cc:1034] running on GCE node
I20260927 18:42:46.290041 27240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:46.290735 27240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:46.291908 27240 hybrid_clock.cc:648] HybridClock initialized: now 1790534566291888 us; error 39 us; skew 500 ppm
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:46.294328 27240 init.cc:377] Logged in from keytab as oryx/127.18.235.131@KRBTEST.COM (short username oryx)
I20260927 18:42:46.295197 27240 webserver.cc:460] Webserver started at http://127.18.235.131:45457/ using document root <none> and password file <none>
I20260927 18:42:46.295691 27240 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:46.295746 27240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:46.295933 27240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:46.297798 27240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "36ee887bb35348e08ee022e679098bcd"
format_stamp: "Formatted at 2026-09-27 18:42:46 on dist-test-slave-6lr6"
server_key: "53b0be6b2a071e552a084b1a0b9c8b79"
server_key_iv: "ae755cef87035046505d0a0e1df19ed9"
server_key_version: "encryptionkey@0"
I20260927 18:42:46.298287 27240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "36ee887bb35348e08ee022e679098bcd"
format_stamp: "Formatted at 2026-09-27 18:42:46 on dist-test-slave-6lr6"
server_key: "53b0be6b2a071e552a084b1a0b9c8b79"
server_key_iv: "ae755cef87035046505d0a0e1df19ed9"
server_key_version: "encryptionkey@0"
I20260927 18:42:46.301554 27240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:42:46.303598 27255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:46.304678 27240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:46.304765 27240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "36ee887bb35348e08ee022e679098bcd"
format_stamp: "Formatted at 2026-09-27 18:42:46 on dist-test-slave-6lr6"
server_key: "53b0be6b2a071e552a084b1a0b9c8b79"
server_key_iv: "ae755cef87035046505d0a0e1df19ed9"
server_key_version: "encryptionkey@0"
I20260927 18:42:46.304858 27240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:46.341887 27240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:46.344447 27240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:46.344609 27240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:46.345980 27240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:46.346026 27240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:46.346065 27240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:46.346093 27240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:46.355548 27240 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:46069
I20260927 18:42:46.355834 27370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:46069 every 8 connection(s)
I20260927 18:42:46.356453 27240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:46.358366 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.346694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:51101 (local address 127.18.235.190:41329)
0927 18:42:46.346845 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.346850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.347592 (+ 742us) server_negotiation.cc:407] Connection header received
0927 18:42:46.348697 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.348702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.348759 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.348858 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.350027 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.350806 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.351938 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.352165 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.352252 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.354807 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.354810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.354812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.354836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.356252 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.356784 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.356787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.356789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.356939 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.357427 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.357430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.357432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.357631 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.357735 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.358166 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.358255 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:42:46.359016 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.347023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.18.235.131:51101)
0927 18:42:46.347439 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.347485 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.348423 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.349029 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.349038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.349436 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:46.349903 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.349916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.350931 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.350935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.351837 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.351848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.352310 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.352314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.352464 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.352960 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:46.352978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:46.354640 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:46.356353 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.356359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.356372 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.356674 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.357134 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.357137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.357140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.357323 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.357735 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:46.357741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:46.357978 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.358117 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.358329 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
I20260927 18:42:46.359985 27371 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41329
I20260927 18:42:46.360107 27371 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:46.360599 27371 heartbeater.cc:511] Master 127.18.235.190:41329 requested a full tablet report, sending...
I20260927 18:42:46.361568 26887 ts_manager.cc:194] Registered new tserver with Master: 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:46.362212 26887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:51101
I20260927 18:42:46.363039 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 27240
I20260927 18:42:46.363163 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:46.363435 19374 external_mini_cluster.cc:1428] Setting key 799a9441002d347f0022613021b6a153
I20260927 18:42:46.365319 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:46.368530 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.362017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:55255 (local address 127.18.235.190:41329)
0927 18:42:46.362110 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.362114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.362254 (+ 140us) server_negotiation.cc:407] Connection header received
0927 18:42:46.362403 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.362406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.362447 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.362522 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.362948 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.363612 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.364696 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.364991 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.365081 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.366020 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.366024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.366027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.366049 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.367236 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.367581 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.367584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.367586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.367651 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.367873 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.367877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.367879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.368020 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.368097 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.368349 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.368434 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:42:46.368559 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.361958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.18.235.131:55255)
0927 18:42:46.362184 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.362195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.362274 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.362558 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.362561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.362710 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:46.362839 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.362847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.363915 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.363918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.364609 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.364618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.365127 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.365131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.365219 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.365620 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:46.365630 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:46.365898 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:46.367329 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.367332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.367334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.367503 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.367724 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.367727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.367729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.367801 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.368100 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:46.368103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:46.368182 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.368282 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.368394 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:42:46.386950 26937 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:46.381699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.18.235.190:41329)
0927 18:42:46.381816 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.381820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.381829 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:42:46.381856 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.381858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.381895 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.381955 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.382580 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.383049 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.383833 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.384035 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.384118 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.386843 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55786: BlockingRecv error: recv got EOF from 127.0.0.1:55786 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:46.395821 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.388260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.18.235.190:41329)
0927 18:42:46.388433 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.388436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.388492 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:42:46.388609 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.388612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.388648 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.388708 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.389269 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.389995 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.390798 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.391010 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.391097 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.393353 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.393357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.393360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.393384 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.394653 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.394992 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.394995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.394996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.395056 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.395229 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.395231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.395233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.395363 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.395464 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.395600 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.395675 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 18:42:46.398229 26887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:46.401007 26887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:46.413435 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.408099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:39105 (local address 127.0.0.1:54580)
0927 18:42:46.408543 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.408590 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.408711 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.409637 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.409642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.409663 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.409792 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.409800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.410945 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.410948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.411952 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.411961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.412960 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.412963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.413036 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.413061 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.413192 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.413306 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":18,"spinlock_wait_cycles":2304,"thread_start_us":76,"threads_started":1}
I20260927 18:42:46.413451 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.408226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.18.235.130:39105)
0927 18:42:46.409000 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.409012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.409034 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:46.409124 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.409130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.409343 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.409490 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.409913 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.410835 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.412088 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.412863 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.412999 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.413145 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.413220 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.413306 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":102,"threads_started":1}
I20260927 18:42:46.414018 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.408774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46069 (local address 127.0.0.1:42564)
0927 18:42:46.409425 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.409438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.409587 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.410112 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.410115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.410127 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.410268 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.410275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.411371 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.411374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.412650 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.412661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.413658 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.413661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.413753 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.413774 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.413827 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.413894 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":126,"thread_start_us":88,"threads_started":1}
I20260927 18:42:46.414070 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.408935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.18.235.131:46069)
0927 18:42:46.409292 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.409301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.409519 (+ 218us) server_negotiation.cc:407] Connection header received
0927 18:42:46.409715 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.409720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.409858 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.409965 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.410403 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.411237 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.412768 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.413560 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.413685 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.413797 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.413858 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.413946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":92,"threads_started":1}
I20260927 18:42:46.414942 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.408197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43835 (local address 127.0.0.1:49666)
0927 18:42:46.408566 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.408590 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.408711 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.410095 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.410098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.410115 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.410270 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.410275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.411362 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.411365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.412650 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.412661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.414571 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.414575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.414668 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.414687 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.414758 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.414822 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":78,"spinlock_wait_cycles":27136,"thread_start_us":69,"threads_started":1}
I20260927 18:42:46.416311 27165 tablet_service.cc:1467] Processing CreateTablet for tablet 67a6fc5466cb4e81a2949722f06f6d15 (DEFAULT_TABLE table=test-table [id=37952bcecab7409fbc6f7fb019af4894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:46.416415 27305 tablet_service.cc:1467] Processing CreateTablet for tablet 67a6fc5466cb4e81a2949722f06f6d15 (DEFAULT_TABLE table=test-table [id=37952bcecab7409fbc6f7fb019af4894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:46.417244 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a6fc5466cb4e81a2949722f06f6d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.417240 27165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a6fc5466cb4e81a2949722f06f6d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.417433 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.409282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.18.235.129:43835)
0927 18:42:46.409541 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.409553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.409571 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:42:46.409638 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.409643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.409823 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.409966 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.410384 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.411254 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.413183 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.414472 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.414590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.417169 (+ 2579us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.417218 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.417304 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260927 18:42:46.419780 27025 tablet_service.cc:1467] Processing CreateTablet for tablet 67a6fc5466cb4e81a2949722f06f6d15 (DEFAULT_TABLE table=test-table [id=37952bcecab7409fbc6f7fb019af4894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:46.420653 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a6fc5466cb4e81a2949722f06f6d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.424150 27399 tablet_bootstrap.cc:492] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Bootstrap starting.
I20260927 18:42:46.424760 27400 tablet_bootstrap.cc:492] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Bootstrap starting.
I20260927 18:42:46.426077 27399 tablet_bootstrap.cc:654] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.426734 27399 log.cc:824] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:46.426781 27400 tablet_bootstrap.cc:654] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.427215 27401 tablet_bootstrap.cc:492] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Bootstrap starting.
I20260927 18:42:46.427539 27400 log.cc:824] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:46.429307 27401 tablet_bootstrap.cc:654] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.429364 27400 tablet_bootstrap.cc:492] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: No bootstrap required, opened a new log
I20260927 18:42:46.429364 27399 tablet_bootstrap.cc:492] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: No bootstrap required, opened a new log
I20260927 18:42:46.429529 27400 ts_tablet_manager.cc:1397] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:42:46.429529 27399 ts_tablet_manager.cc:1397] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:42:46.430032 27401 log.cc:824] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:46.431542 27401 tablet_bootstrap.cc:492] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: No bootstrap required, opened a new log
I20260927 18:42:46.431697 27401 ts_tablet_manager.cc:1397] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:42:46.433277 27399 raft_consensus.cc:348] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.433277 27400 raft_consensus.cc:348] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.433522 27400 raft_consensus.cc:374] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.433522 27399 raft_consensus.cc:374] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.433576 27399 raft_consensus.cc:729] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.433576 27400 raft_consensus.cc:729] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.434038 27400 consensus_queue.cc:260] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.434038 27399 consensus_queue.cc:260] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.434669 27401 raft_consensus.cc:348] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.434846 27231 heartbeater.cc:503] Master 127.18.235.190:41329 was elected leader, sending a full tablet report...
I20260927 18:42:46.434895 27401 raft_consensus.cc:374] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.434937 27401 raft_consensus.cc:729] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7932d91d4334ee6b84eee9cb6d96de5, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.434957 27400 ts_tablet_manager.cc:1428] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:46.434957 27399 ts_tablet_manager.cc:1428] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:42:46.435307 27401 consensus_queue.cc:260] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.435840 27091 heartbeater.cc:503] Master 127.18.235.190:41329 was elected leader, sending a full tablet report...
I20260927 18:42:46.436214 27401 ts_tablet_manager.cc:1428] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 18:42:46.436698 27371 heartbeater.cc:503] Master 127.18.235.190:41329 was elected leader, sending a full tablet report...
I20260927 18:42:46.438483 27406 raft_consensus.cc:482] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:46.438612 27406 raft_consensus.cc:504] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.439796 27406 leader_election.cc:288] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105), a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835)
I20260927 18:42:46.443712 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.439748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:39105 (local address 127.18.235.131:42549)
0927 18:42:46.439854 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.439878 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.439953 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.440370 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.440372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.440388 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.440504 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.440510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.441477 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.441480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.442732 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.442743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.443403 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.443406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.443486 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.443505 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.443558 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.443610 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":19,"spinlock_wait_cycles":15232}
I20260927 18:42:46.443749 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.439954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:42549 (local address 127.18.235.130:39105)
0927 18:42:46.440067 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.440071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.440083 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:46.440121 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.440123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.440155 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.440233 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.440602 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.441391 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.442846 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.443307 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.443411 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.443456 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.443582 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.443642 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:42:46.444113 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.439905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:56159 (local address 127.18.235.129:43835)
0927 18:42:46.440003 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.440007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.440409 (+ 402us) server_negotiation.cc:407] Connection header received
0927 18:42:46.440509 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.440511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.440543 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.440613 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.440815 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.441537 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.443137 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.443643 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.443751 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.443793 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.443955 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.444017 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:42:46.444131 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.439772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43835 (local address 127.18.235.131:56159)
0927 18:42:46.440327 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.440340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.440437 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.440627 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.440630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.440645 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.440732 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.440736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.441612 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.441615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.443044 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.443051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.443758 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.443762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.443853 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.443875 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.443940 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.444001 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":291,"spinlock_wait_cycles":2176,"thread_start_us":72,"threads_started":1}
I20260927 18:42:46.444332 27185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6fc5466cb4e81a2949722f06f6d15" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" is_pre_election: true
I20260927 18:42:46.444591 27185 raft_consensus.cc:2454] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 0.
I20260927 18:42:46.444828 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6fc5466cb4e81a2949722f06f6d15" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" is_pre_election: true
I20260927 18:42:46.445051 27045 raft_consensus.cc:2454] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 0.
I20260927 18:42:46.445034 27256 leader_election.cc:302] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ee887bb35348e08ee022e679098bcd, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:46.445269 27406 raft_consensus.cc:2781] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:46.445336 27406 raft_consensus.cc:482] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:46.445406 27406 raft_consensus.cc:3037] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.446270 27406 raft_consensus.cc:504] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.446558 27406 leader_election.cc:288] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 election: Requested vote from peers e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105), a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835)
I20260927 18:42:46.446816 27185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6fc5466cb4e81a2949722f06f6d15" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dfc6f35ca54a8bab6c092dd1859678"
I20260927 18:42:46.446847 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6fc5466cb4e81a2949722f06f6d15" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
I20260927 18:42:46.446921 27185 raft_consensus.cc:3037] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.446942 27045 raft_consensus.cc:3037] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.447781 27185 raft_consensus.cc:2454] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 1.
I20260927 18:42:46.447798 27045 raft_consensus.cc:2454] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 1.
I20260927 18:42:46.448024 27256 leader_election.cc:302] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ee887bb35348e08ee022e679098bcd, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:46.448175 27406 raft_consensus.cc:2781] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:46.448345 27406 raft_consensus.cc:686] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 1 LEADER]: Becoming Leader. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Running, Role: LEADER
I20260927 18:42:46.448603 27406 consensus_queue.cc:237] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.451049 26887 catalog_manager.cc:5495] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ee887bb35348e08ee022e679098bcd (127.18.235.131). New cstate: current_term: 1 leader_uuid: "36ee887bb35348e08ee022e679098bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } health_report { overall_health: HEALTHY } } }
I20260927 18:42:46.461238 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.458391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.18.235.131:46069)
0927 18:42:46.458489 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.458492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.458503 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:46.458600 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.458602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.458641 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.458713 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:46.458867 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.459297 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.460085 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.460326 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.460394 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.460564 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:46.460900 (+ 336us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:46.460982 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.461044 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.461083 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:42:46.465770 27045 raft_consensus.cc:1261] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Refusing update from remote peer 36ee887bb35348e08ee022e679098bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:46.466030 27185 raft_consensus.cc:1261] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Refusing update from remote peer 36ee887bb35348e08ee022e679098bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:46.466315 27406 consensus_queue.cc:1035] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:46.466487 27409 consensus_queue.cc:1035] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:46.474506 27414 mvcc.cc:205] Tried to move back new op lower bound from 7334029584238288896 to 7334029584175755264. Current Snapshot: MvccSnapshot[applied={T|T < 7334029584238288896}]
I20260927 18:42:46.480228 26887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:46.480373 26887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:46.482492 26887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Sending DeleteTablet for 3 replicas of tablet 67a6fc5466cb4e81a2949722f06f6d15
I20260927 18:42:46.483141 27165 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a6fc5466cb4e81a2949722f06f6d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:46 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:54580
I20260927 18:42:46.483247 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a6fc5466cb4e81a2949722f06f6d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:46 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49666
I20260927 18:42:46.483696 27422 tablet_replica.cc:330] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: stopping tablet replica
I20260927 18:42:46.483675 27305 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a6fc5466cb4e81a2949722f06f6d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:46 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 18:42:46.483973 27422 raft_consensus.cc:2229] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:46.484081 27423 tablet_replica.cc:330] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: stopping tablet replica
I20260927 18:42:46.484058 26887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:42:46.484226 27422 raft_consensus.cc:2258] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 18:42:46.484447 26887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:46.484581 27423 raft_consensus.cc:2229] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:46.484948 27423 raft_consensus.cc:2258] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:46.485945 27422 ts_tablet_manager.cc:1905] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:46.486222 27423 ts_tablet_manager.cc:1905] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:46.486860 27421 tablet_replica.cc:330] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: stopping tablet replica
I20260927 18:42:46.487109 27421 raft_consensus.cc:2229] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:46.487366 27421 raft_consensus.cc:2258] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:46.489253 27025 tablet_service.cc:1467] Processing CreateTablet for tablet 04434e6dd6194dfead1e7b22f11a3ae1 (DEFAULT_TABLE table=test-table [id=4a122204acf34fdd819d6329ca03e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:46.489302 27423 ts_tablet_manager.cc:1918] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:46.489421 27305 tablet_service.cc:1467] Processing CreateTablet for tablet 04434e6dd6194dfead1e7b22f11a3ae1 (DEFAULT_TABLE table=test-table [id=4a122204acf34fdd819d6329ca03e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:46.489521 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04434e6dd6194dfead1e7b22f11a3ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.489483 27423 log.cc:1196] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/67a6fc5466cb4e81a2949722f06f6d15
I20260927 18:42:46.489632 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04434e6dd6194dfead1e7b22f11a3ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.489943 27423 ts_tablet_manager.cc:1939] T 67a6fc5466cb4e81a2949722f06f6d15 P 36ee887bb35348e08ee022e679098bcd: Deleting consensus metadata
I20260927 18:42:46.491511 27400 tablet_bootstrap.cc:492] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Bootstrap starting.
I20260927 18:42:46.491627 27401 tablet_bootstrap.cc:492] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Bootstrap starting.
I20260927 18:42:46.491612 26873 catalog_manager.cc:4841] TS 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069): tablet 67a6fc5466cb4e81a2949722f06f6d15 (table test-table [id=37952bcecab7409fbc6f7fb019af4894]) successfully deleted
I20260927 18:42:46.492022 27165 tablet_service.cc:1467] Processing CreateTablet for tablet 04434e6dd6194dfead1e7b22f11a3ae1 (DEFAULT_TABLE table=test-table [id=4a122204acf34fdd819d6329ca03e51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:46.492235 27165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04434e6dd6194dfead1e7b22f11a3ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.492375 27400 tablet_bootstrap.cc:654] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.492460 27401 tablet_bootstrap.cc:654] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.493381 27421 ts_tablet_manager.cc:1905] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:46.493844 27400 tablet_bootstrap.cc:492] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: No bootstrap required, opened a new log
I20260927 18:42:46.493928 27400 ts_tablet_manager.cc:1397] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:46.494328 27400 raft_consensus.cc:348] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.494418 27400 raft_consensus.cc:374] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.494449 27400 raft_consensus.cc:729] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.494542 27399 tablet_bootstrap.cc:492] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Bootstrap starting.
I20260927 18:42:46.494551 27400 consensus_queue.cc:260] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.494793 27400 ts_tablet_manager.cc:1428] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:46.495388 27399 tablet_bootstrap.cc:654] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.495579 27421 ts_tablet_manager.cc:1918] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:46.495667 27421 log.cc:1196] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/67a6fc5466cb4e81a2949722f06f6d15
I20260927 18:42:46.495935 27421 ts_tablet_manager.cc:1939] T 67a6fc5466cb4e81a2949722f06f6d15 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting consensus metadata
I20260927 18:42:46.496906 27399 tablet_bootstrap.cc:492] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: No bootstrap required, opened a new log
I20260927 18:42:46.496914 26872 catalog_manager.cc:4841] TS e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105): tablet 67a6fc5466cb4e81a2949722f06f6d15 (table test-table [id=37952bcecab7409fbc6f7fb019af4894]) successfully deleted
I20260927 18:42:46.496981 27399 ts_tablet_manager.cc:1397] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:46.497396 27399 raft_consensus.cc:348] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.497490 27399 raft_consensus.cc:374] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.497520 27399 raft_consensus.cc:729] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.497635 27399 consensus_queue.cc:260] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.497848 27399 ts_tablet_manager.cc:1428] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:42:46.498008 27422 ts_tablet_manager.cc:1918] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:46.498095 27422 log.cc:1196] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/67a6fc5466cb4e81a2949722f06f6d15
I20260927 18:42:46.498358 27422 ts_tablet_manager.cc:1939] T 67a6fc5466cb4e81a2949722f06f6d15 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting consensus metadata
I20260927 18:42:46.499298 26873 catalog_manager.cc:4841] TS a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835): tablet 67a6fc5466cb4e81a2949722f06f6d15 (table test-table [id=37952bcecab7409fbc6f7fb019af4894]) successfully deleted
I20260927 18:42:46.499753 27401 tablet_bootstrap.cc:492] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: No bootstrap required, opened a new log
I20260927 18:42:46.499828 27401 ts_tablet_manager.cc:1397] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 18:42:46.500357 27401 raft_consensus.cc:348] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.500449 27401 raft_consensus.cc:374] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.500478 27401 raft_consensus.cc:729] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7932d91d4334ee6b84eee9cb6d96de5, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.500582 27401 consensus_queue.cc:260] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.500808 27401 ts_tablet_manager.cc:1428] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 18:42:46.607267 27372 tablet.cc:2369] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:46.637777 27232 tablet.cc:2369] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:46.648126 27405 raft_consensus.cc:482] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:46.648233 27405 raft_consensus.cc:504] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.649219 27405 leader_election.cc:288] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:46.652725 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.649535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:43647 (local address 127.18.235.129:43835)
0927 18:42:46.649644 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.649647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.649711 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:42:46.649897 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.649901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.649950 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.650029 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.650298 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.650861 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.651865 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.652248 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.652344 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.652396 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.652555 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.652620 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:42:46.652750 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.649418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43835 (local address 127.18.235.130:43647)
0927 18:42:46.649642 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.649655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.649754 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.650043 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.650047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.650067 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.650197 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.650202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.650961 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.650965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.651752 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.651760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.652344 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.652347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.652445 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.652468 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.652530 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.652598 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"spinlock_wait_cycles":29952,"thread_start_us":80,"threads_started":1}
I20260927 18:42:46.652845 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.649854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:41417 (local address 127.18.235.131:46069)
0927 18:42:46.649964 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.649968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.650096 (+ 128us) server_negotiation.cc:407] Connection header received
0927 18:42:46.650220 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.650223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.650267 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.650327 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:46.650568 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.651044 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.652000 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.652369 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.652471 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.652520 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.652672 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.652736 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:42:46.652874 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.649763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46069 (local address 127.18.235.130:41417)
0927 18:42:46.650013 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.650023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.650125 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.650347 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.650350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.650364 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:46.650470 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.650475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.651145 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.651148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.651878 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.651886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.652471 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.652474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.652563 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.652585 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.652647 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.652713 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"spinlock_wait_cycles":768,"thread_start_us":63,"threads_started":1}
I20260927 18:42:46.653353 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04434e6dd6194dfead1e7b22f11a3ae1" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" is_pre_election: true
I20260927 18:42:46.653354 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04434e6dd6194dfead1e7b22f11a3ae1" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ee887bb35348e08ee022e679098bcd" is_pre_election: true
I20260927 18:42:46.653497 27045 raft_consensus.cc:2454] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 0.
I20260927 18:42:46.653548 27325 raft_consensus.cc:2454] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 0.
I20260927 18:42:46.653828 27117 leader_election.cc:302] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7932d91d4334ee6b84eee9cb6d96de5, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:46.654047 27405 raft_consensus.cc:2781] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:46.654101 27405 raft_consensus.cc:482] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:46.654141 27405 raft_consensus.cc:3037] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.655085 27405 raft_consensus.cc:504] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.655377 27405 leader_election.cc:288] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 election: Requested vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:46.655675 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04434e6dd6194dfead1e7b22f11a3ae1" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
I20260927 18:42:46.655752 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04434e6dd6194dfead1e7b22f11a3ae1" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ee887bb35348e08ee022e679098bcd"
I20260927 18:42:46.655787 27045 raft_consensus.cc:3037] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.655864 27325 raft_consensus.cc:3037] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.656641 27045 raft_consensus.cc:2454] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 1.
I20260927 18:42:46.656714 27325 raft_consensus.cc:2454] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 1.
I20260927 18:42:46.656872 27117 leader_election.cc:302] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7932d91d4334ee6b84eee9cb6d96de5, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:46.657014 27405 raft_consensus.cc:2781] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:46.657179 27405 raft_consensus.cc:686] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 LEADER]: Becoming Leader. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Running, Role: LEADER
I20260927 18:42:46.657388 27405 consensus_queue.cc:237] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.659198 26887 catalog_manager.cc:5495] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130). New cstate: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } health_report { overall_health: UNKNOWN } } }
W20260927 18:42:46.670284 27092 tablet.cc:2369] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534565, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.190@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:46.713711 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.706480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41329 (local address 127.0.0.1:55800)
0927 18:42:46.706700 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.706713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.706815 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.707014 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.707017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.707413 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:46.707500 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.707506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.708338 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.708340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.708913 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.708921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.709328 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.709331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.709387 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.709834 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:46.709845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:46.711161 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:46.712457 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.712464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.712469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.712676 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.712872 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:46.712874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:46.712875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:46.712941 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:46.713263 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:46.713268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:46.713363 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.713449 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.713556 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260927 18:42:46.713726 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.706540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.18.235.190:41329)
0927 18:42:46.706643 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.706648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.706774 (+ 126us) server_negotiation.cc:407] Connection header received
0927 18:42:46.706886 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.706889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.706926 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.706985 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.707682 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.708242 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.709003 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.709250 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.709323 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.711287 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.711290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.711292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.711313 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.712371 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.712751 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.712754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.712755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.712816 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.712996 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.712997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.712999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.713188 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.713259 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.713488 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.713607 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":34}
I20260927 18:42:46.717068 26887 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:55800:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:42:46.717757 26887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:46.721932 27025 tablet_service.cc:1467] Processing CreateTablet for tablet fb3f0e7f2c0a43fe8699bc94c5630344 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=853b3086b8c441d3bd2a104397386ec7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:46.721976 27165 tablet_service.cc:1467] Processing CreateTablet for tablet fb3f0e7f2c0a43fe8699bc94c5630344 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=853b3086b8c441d3bd2a104397386ec7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:46.722309 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3f0e7f2c0a43fe8699bc94c5630344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.722311 27165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3f0e7f2c0a43fe8699bc94c5630344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.722254 27305 tablet_service.cc:1467] Processing CreateTablet for tablet fb3f0e7f2c0a43fe8699bc94c5630344 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=853b3086b8c441d3bd2a104397386ec7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:42:46.722564 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3f0e7f2c0a43fe8699bc94c5630344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:46.724628 27399 tablet_bootstrap.cc:492] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: Bootstrap starting.
I20260927 18:42:46.724968 27401 tablet_bootstrap.cc:492] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: Bootstrap starting.
I20260927 18:42:46.725008 27400 tablet_bootstrap.cc:492] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: Bootstrap starting.
I20260927 18:42:46.725613 27399 tablet_bootstrap.cc:654] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.725782 27400 tablet_bootstrap.cc:654] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.725855 27401 tablet_bootstrap.cc:654] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:46.726804 27400 tablet_bootstrap.cc:492] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: No bootstrap required, opened a new log
I20260927 18:42:46.726871 27401 tablet_bootstrap.cc:492] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: No bootstrap required, opened a new log
I20260927 18:42:46.726890 27400 ts_tablet_manager.cc:1397] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:46.726948 27401 ts_tablet_manager.cc:1397] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:46.727029 27399 tablet_bootstrap.cc:492] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: No bootstrap required, opened a new log
I20260927 18:42:46.727106 27399 ts_tablet_manager.cc:1397] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 18:42:46.727267 27400 raft_consensus.cc:348] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.727357 27400 raft_consensus.cc:374] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.727388 27400 raft_consensus.cc:729] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.727353 27401 raft_consensus.cc:348] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.727445 27401 raft_consensus.cc:374] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.727475 27401 raft_consensus.cc:729] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7932d91d4334ee6b84eee9cb6d96de5, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.727488 27399 raft_consensus.cc:348] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.727504 27400 consensus_queue.cc:260] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.727572 27399 raft_consensus.cc:374] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:46.727602 27399 raft_consensus.cc:729] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Initialized, Role: FOLLOWER
I20260927 18:42:46.727582 27401 consensus_queue.cc:260] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.727699 27399 consensus_queue.cc:260] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.727762 27400 ts_tablet_manager.cc:1428] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:46.727830 27401 ts_tablet_manager.cc:1428] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:46.727916 27399 ts_tablet_manager.cc:1428] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:46.727979 27409 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.728024 27407 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.728098 27409 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.728082 27405 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.728138 27407 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.728169 27405 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.856482 27405 raft_consensus.cc:482] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:46.856591 27405 raft_consensus.cc:504] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.856936 27405 leader_election.cc:288] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:46.857370 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3f0e7f2c0a43fe8699bc94c5630344" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" is_pre_election: true
I20260927 18:42:46.857371 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3f0e7f2c0a43fe8699bc94c5630344" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ee887bb35348e08ee022e679098bcd" is_pre_election: true
I20260927 18:42:46.857515 27045 raft_consensus.cc:2454] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 0.
I20260927 18:42:46.857515 27325 raft_consensus.cc:2454] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 0.
I20260927 18:42:46.857777 27117 leader_election.cc:302] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ee887bb35348e08ee022e679098bcd, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:46.857960 27405 raft_consensus.cc:2781] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:46.858021 27405 raft_consensus.cc:482] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:46.858047 27405 raft_consensus.cc:3037] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.858963 27405 raft_consensus.cc:504] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.859258 27405 leader_election.cc:288] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 election: Requested vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:46.859536 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3f0e7f2c0a43fe8699bc94c5630344" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
I20260927 18:42:46.859606 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3f0e7f2c0a43fe8699bc94c5630344" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ee887bb35348e08ee022e679098bcd"
I20260927 18:42:46.859680 27045 raft_consensus.cc:3037] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.859709 27325 raft_consensus.cc:3037] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:46.860728 27045 raft_consensus.cc:2454] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 1.
I20260927 18:42:46.860742 27325 raft_consensus.cc:2454] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 1.
I20260927 18:42:46.860987 27117 leader_election.cc:302] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7932d91d4334ee6b84eee9cb6d96de5, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:46.861168 27405 raft_consensus.cc:2781] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:46.861251 27405 raft_consensus.cc:686] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 LEADER]: Becoming Leader. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Running, Role: LEADER
I20260927 18:42:46.861389 27405 consensus_queue.cc:237] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:46.862037 27429 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: TxnStatusTablet state changed. Reason: New leader e6dfc6f35ca54a8bab6c092dd1859678. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.862129 27429 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:46.862416 27441 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:42:46.862572 27441 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:42:46.862943 26886 catalog_manager.cc:5495] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130). New cstate: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:46.879204 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.875721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.18.235.130:39105)
0927 18:42:46.875830 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.875834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.875845 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:46.876008 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.876010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.876056 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.876127 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:46.876353 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.876829 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.878006 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.878209 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.878279 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.878549 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:46.878916 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:46.878992 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.879072 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.879116 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:42:46.879273 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.875631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:39105 (local address 127.0.0.1:54594)
0927 18:42:46.875783 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.875799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.875895 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.876149 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.876151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.876158 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:46.876274 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.876280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.876912 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.876916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.877907 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.877918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.878294 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.878297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.878389 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.878439 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:46.878990 (+ 551us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:46.879001 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.879064 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.879146 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 18:42:46.881438 27045 raft_consensus.cc:1261] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Refusing update from remote peer e6dfc6f35ca54a8bab6c092dd1859678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:46.881620 27325 raft_consensus.cc:1261] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Refusing update from remote peer e6dfc6f35ca54a8bab6c092dd1859678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:46.881827 27405 consensus_queue.cc:1035] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:46.882081 27429 consensus_queue.cc:1035] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:46.883458 27409 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: TxnStatusTablet state changed. Reason: New leader e6dfc6f35ca54a8bab6c092dd1859678. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.883559 27409 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.883534 27407 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: TxnStatusTablet state changed. Reason: New leader e6dfc6f35ca54a8bab6c092dd1859678. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.883628 27407 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.885768 27429 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.885876 27429 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:46.885886 27446 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.886077 27446 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P e6dfc6f35ca54a8bab6c092dd1859678: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:42:46.886916 27407 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.886994 27413 mvcc.cc:205] Tried to move back new op lower bound from 7334029585942446080 to 7334029585865527296. Current Snapshot: MvccSnapshot[applied={T|T < 7334029585942446080}]
I20260927 18:42:46.887013 27407 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P a7932d91d4334ee6b84eee9cb6d96de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.887710 27409 tablet_replica.cc:439] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } } }
I20260927 18:42:46.887799 27409 tablet_replica.cc:442] T fb3f0e7f2c0a43fe8699bc94c5630344 P 36ee887bb35348e08ee022e679098bcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:42:46.887956 27412 mvcc.cc:205] Tried to move back new op lower bound from 7334029585942446080 to 7334029585865527296. Current Snapshot: MvccSnapshot[applied={T|T < 7334029585942446080}]
I20260927 18:42:46.894191 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.891370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.18.235.130:39105)
0927 18:42:46.891467 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.891472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.891485 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:42:46.891522 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.891525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.891569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.891641 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:46.891897 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.892383 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.893199 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.893450 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.893546 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.893787 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:46.893857 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:46.893920 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.894014 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.894060 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 18:42:46.903366 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.900328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:36613 (local address 127.18.235.130:39105)
0927 18:42:46.900424 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.900429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.900611 (+ 182us) server_negotiation.cc:407] Connection header received
0927 18:42:46.900743 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.900746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.900789 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.900851 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:46.901090 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.901566 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.902392 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.902604 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.902704 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.902971 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:46.903029 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:46.903089 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.903194 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.903231 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":32}
I20260927 18:42:46.903401 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.900273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:39105 (local address 127.18.235.130:36613)
0927 18:42:46.900539 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:46.900550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:46.900655 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:46.900864 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:46.900867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:46.900886 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:42:46.900998 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.901003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.901668 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.901671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.902285 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:46.902293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.902703 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.902707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:46.902811 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.902848 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:42:46.903103 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:42:46.903109 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:42:46.903184 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:42:46.903253 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260927 18:42:46.909818 27325 raft_consensus.cc:1261] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Refusing update from remote peer e6dfc6f35ca54a8bab6c092dd1859678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:46.909844 27045 raft_consensus.cc:1261] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Refusing update from remote peer e6dfc6f35ca54a8bab6c092dd1859678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:46.910181 27446 consensus_queue.cc:1035] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:46.910328 27405 consensus_queue.cc:1035] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:46.951048 26887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:46.951170 26887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:46.952989 26887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Sending DeleteTablet for 3 replicas of tablet 04434e6dd6194dfead1e7b22f11a3ae1
I20260927 18:42:46.953493 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet 04434e6dd6194dfead1e7b22f11a3ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:46 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49666
I20260927 18:42:46.953672 27422 tablet_replica.cc:330] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: stopping tablet replica
I20260927 18:42:46.953773 27422 raft_consensus.cc:2229] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:46.953853 27422 raft_consensus.cc:2258] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:46.954185 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet 04434e6dd6194dfead1e7b22f11a3ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:46 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:54580
I20260927 18:42:46.954356 27421 tablet_replica.cc:330] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: stopping tablet replica
I20260927 18:42:46.954470 27421 raft_consensus.cc:2229] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:46.954720 27421 raft_consensus.cc:2258] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:46.955277 27422 ts_tablet_manager.cc:1905] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:46.957607 27422 ts_tablet_manager.cc:1918] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:46.957682 27422 log.cc:1196] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/04434e6dd6194dfead1e7b22f11a3ae1
I20260927 18:42:46.957868 27422 ts_tablet_manager.cc:1939] T 04434e6dd6194dfead1e7b22f11a3ae1 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting consensus metadata
I20260927 18:42:46.958508 26873 catalog_manager.cc:4841] TS a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835): tablet 04434e6dd6194dfead1e7b22f11a3ae1 (table test-table [id=4a122204acf34fdd819d6329ca03e51b]) successfully deleted
I20260927 18:42:46.958609 27305 tablet_service.cc:1514] Processing DeleteTablet for tablet 04434e6dd6194dfead1e7b22f11a3ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:46 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 18:42:46.958793 27423 tablet_replica.cc:330] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: stopping tablet replica
I20260927 18:42:46.958900 27423 raft_consensus.cc:2229] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:46.958914 27421 ts_tablet_manager.cc:1905] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:46.958992 27423 raft_consensus.cc:2258] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:46.961577 27421 ts_tablet_manager.cc:1918] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:46.961649 27421 log.cc:1196] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/04434e6dd6194dfead1e7b22f11a3ae1
I20260927 18:42:46.961865 27421 ts_tablet_manager.cc:1939] T 04434e6dd6194dfead1e7b22f11a3ae1 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting consensus metadata
I20260927 18:42:46.962481 26872 catalog_manager.cc:4841] TS e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105): tablet 04434e6dd6194dfead1e7b22f11a3ae1 (table test-table [id=4a122204acf34fdd819d6329ca03e51b]) successfully deleted
I20260927 18:42:46.963204 27423 ts_tablet_manager.cc:1905] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:46.965739 27423 ts_tablet_manager.cc:1918] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:46.965812 27423 log.cc:1196] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/04434e6dd6194dfead1e7b22f11a3ae1
I20260927 18:42:46.966019 27423 ts_tablet_manager.cc:1939] T 04434e6dd6194dfead1e7b22f11a3ae1 P 36ee887bb35348e08ee022e679098bcd: Deleting consensus metadata
I20260927 18:42:46.966579 26873 catalog_manager.cc:4841] TS 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069): tablet 04434e6dd6194dfead1e7b22f11a3ae1 (table test-table [id=4a122204acf34fdd819d6329ca03e51b]) successfully deleted
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:42:46.985229 26937 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:46.979260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.18.235.190:41329)
0927 18:42:46.979360 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.979363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.979587 (+ 224us) server_negotiation.cc:407] Connection header received
0927 18:42:46.979709 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.979712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.979752 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.979809 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.980453 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.980965 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.981817 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.982038 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.982148 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.985065 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55810: BlockingRecv error: recv got EOF from 127.0.0.1:55810 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 18:42:46 dist-test-slave-6lr6 krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM
I20260927 18:42:46.994395 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:46.986591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.18.235.190:41329)
0927 18:42:46.986681 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:46.986684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:46.986862 (+ 178us) server_negotiation.cc:407] Connection header received
0927 18:42:46.986985 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:46.986988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:46.987027 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:46.987084 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:46.987594 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.988141 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.988902 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:46.989161 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:46.989239 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:46.991544 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:46.991547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:46.991549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:46.991570 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:46.992821 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.993333 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.993335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.993336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.993413 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:46.993713 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:46.993719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:46.993720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:46.993860 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:46.993954 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:46.994207 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:42:46.994285 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:42:46.996825 26887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:46.997351 26887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:47.001894 27025 tablet_service.cc:1467] Processing CreateTablet for tablet 8f72f1301db3417db86cf5c0128b0ab4 (DEFAULT_TABLE table=test-table [id=65167cd787274580b02d533613e2e3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:47.002054 27305 tablet_service.cc:1467] Processing CreateTablet for tablet 8f72f1301db3417db86cf5c0128b0ab4 (DEFAULT_TABLE table=test-table [id=65167cd787274580b02d533613e2e3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:47.002192 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f72f1301db3417db86cf5c0128b0ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:47.002336 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f72f1301db3417db86cf5c0128b0ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:47.003923 27401 tablet_bootstrap.cc:492] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Bootstrap starting.
I20260927 18:42:47.004103 27164 tablet_service.cc:1467] Processing CreateTablet for tablet 8f72f1301db3417db86cf5c0128b0ab4 (DEFAULT_TABLE table=test-table [id=65167cd787274580b02d533613e2e3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:47.004338 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f72f1301db3417db86cf5c0128b0ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:47.004691 27401 tablet_bootstrap.cc:654] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:47.006335 27399 tablet_bootstrap.cc:492] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Bootstrap starting.
I20260927 18:42:47.006400 27400 tablet_bootstrap.cc:492] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Bootstrap starting.
I20260927 18:42:47.007153 27399 tablet_bootstrap.cc:654] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:47.007153 27400 tablet_bootstrap.cc:654] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:47.008222 27399 tablet_bootstrap.cc:492] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: No bootstrap required, opened a new log
I20260927 18:42:47.008306 27399 ts_tablet_manager.cc:1397] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:47.008430 27400 tablet_bootstrap.cc:492] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: No bootstrap required, opened a new log
I20260927 18:42:47.008524 27400 ts_tablet_manager.cc:1397] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:47.008728 27399 raft_consensus.cc:348] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.008836 27399 raft_consensus.cc:374] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:47.008868 27399 raft_consensus.cc:729] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Initialized, Role: FOLLOWER
I20260927 18:42:47.008977 27399 consensus_queue.cc:260] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.009004 27400 raft_consensus.cc:348] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.009166 27400 raft_consensus.cc:374] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:47.009208 27400 raft_consensus.cc:729] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Initialized, Role: FOLLOWER
I20260927 18:42:47.009272 27399 ts_tablet_manager.cc:1428] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:47.009327 27400 consensus_queue.cc:260] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.009609 27400 ts_tablet_manager.cc:1428] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:47.010028 27401 tablet_bootstrap.cc:492] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: No bootstrap required, opened a new log
I20260927 18:42:47.010108 27401 ts_tablet_manager.cc:1397] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 18:42:47.010691 27401 raft_consensus.cc:348] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.010792 27401 raft_consensus.cc:374] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:47.010821 27401 raft_consensus.cc:729] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7932d91d4334ee6b84eee9cb6d96de5, State: Initialized, Role: FOLLOWER
I20260927 18:42:47.010923 27401 consensus_queue.cc:260] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.011183 27401 ts_tablet_manager.cc:1428] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:47.122946 27409 raft_consensus.cc:482] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:47.123066 27409 raft_consensus.cc:504] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.123477 27409 leader_election.cc:288] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105)
I20260927 18:42:47.123844 27185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f72f1301db3417db86cf5c0128b0ab4" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" is_pre_election: true
I20260927 18:42:47.123854 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f72f1301db3417db86cf5c0128b0ab4" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" is_pre_election: true
I20260927 18:42:47.123991 27185 raft_consensus.cc:2454] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 0.
I20260927 18:42:47.124007 27045 raft_consensus.cc:2454] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 0.
I20260927 18:42:47.124269 27256 leader_election.cc:302] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ee887bb35348e08ee022e679098bcd, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:47.124459 27409 raft_consensus.cc:2781] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:47.124521 27409 raft_consensus.cc:482] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:47.124564 27409 raft_consensus.cc:3037] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:47.125572 27409 raft_consensus.cc:504] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.125880 27409 leader_election.cc:288] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 election: Requested vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105)
I20260927 18:42:47.126180 27185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f72f1301db3417db86cf5c0128b0ab4" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dfc6f35ca54a8bab6c092dd1859678"
I20260927 18:42:47.126195 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f72f1301db3417db86cf5c0128b0ab4" candidate_uuid: "36ee887bb35348e08ee022e679098bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
I20260927 18:42:47.126281 27185 raft_consensus.cc:3037] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:47.126312 27045 raft_consensus.cc:3037] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:47.127079 27185 raft_consensus.cc:2454] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 1.
I20260927 18:42:47.127319 27256 leader_election.cc:302] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ee887bb35348e08ee022e679098bcd, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:47.127461 27045 raft_consensus.cc:2454] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ee887bb35348e08ee022e679098bcd in term 1.
I20260927 18:42:47.127480 27409 raft_consensus.cc:2781] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:47.127655 27409 raft_consensus.cc:686] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 1 LEADER]: Becoming Leader. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Running, Role: LEADER
I20260927 18:42:47.127796 27409 consensus_queue.cc:237] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.129453 26888 catalog_manager.cc:5495] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ee887bb35348e08ee022e679098bcd (127.18.235.131). New cstate: current_term: 1 leader_uuid: "36ee887bb35348e08ee022e679098bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } health_report { overall_health: HEALTHY } } }
I20260927 18:42:47.164815 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:47.162138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.18.235.131:46069)
0927 18:42:47.162345 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:47.162349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:47.162360 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:47.162402 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:47.162404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:47.162445 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:47.162518 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:47.162707 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:47.163207 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:47.163984 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:47.164213 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:47.164291 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:47.164482 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:47.164544 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:47.164589 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:47.164646 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 18:42:47.164705 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
I20260927 18:42:47.166561 27185 raft_consensus.cc:1261] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Refusing update from remote peer 36ee887bb35348e08ee022e679098bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:47.166575 27045 raft_consensus.cc:1261] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Refusing update from remote peer 36ee887bb35348e08ee022e679098bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:47.166929 27409 consensus_queue.cc:1035] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:47.167115 27468 consensus_queue.cc:1035] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:47.173879 26888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:47.174031 26888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:47.175849 26888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Sending DeleteTablet for 3 replicas of tablet 8f72f1301db3417db86cf5c0128b0ab4
I20260927 18:42:47.176295 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f72f1301db3417db86cf5c0128b0ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:47 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:54580
I20260927 18:42:47.176309 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f72f1301db3417db86cf5c0128b0ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:47 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49666
I20260927 18:42:47.176455 27421 tablet_replica.cc:330] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: stopping tablet replica
I20260927 18:42:47.176468 27305 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f72f1301db3417db86cf5c0128b0ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:47 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 18:42:47.176558 27421 raft_consensus.cc:2229] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:47.176633 27422 tablet_replica.cc:330] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: stopping tablet replica
I20260927 18:42:47.176674 27421 raft_consensus.cc:2258] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:47.176729 27422 raft_consensus.cc:2229] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:47.176837 27422 raft_consensus.cc:2258] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:47.176985 27423 tablet_replica.cc:330] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: stopping tablet replica
I20260927 18:42:47.177093 27423 raft_consensus.cc:2229] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:47.177052 26888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:42:47.177277 27421 ts_tablet_manager.cc:1905] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:47.177276 27423 raft_consensus.cc:2258] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 18:42:47.177449 26888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:47.177771 27422 ts_tablet_manager.cc:1905] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:47.179601 27421 ts_tablet_manager.cc:1918] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:47.179680 27421 log.cc:1196] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/8f72f1301db3417db86cf5c0128b0ab4
I20260927 18:42:47.179874 27422 ts_tablet_manager.cc:1918] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:47.179939 27421 ts_tablet_manager.cc:1939] T 8f72f1301db3417db86cf5c0128b0ab4 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting consensus metadata
I20260927 18:42:47.179947 27422 log.cc:1196] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/8f72f1301db3417db86cf5c0128b0ab4
I20260927 18:42:47.180145 27422 ts_tablet_manager.cc:1939] T 8f72f1301db3417db86cf5c0128b0ab4 P a7932d91d4334ee6b84eee9cb6d96de5: Deleting consensus metadata
I20260927 18:42:47.180531 26872 catalog_manager.cc:4841] TS e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105): tablet 8f72f1301db3417db86cf5c0128b0ab4 (table test-table [id=65167cd787274580b02d533613e2e3cd]) successfully deleted
I20260927 18:42:47.180665 26873 catalog_manager.cc:4841] TS a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835): tablet 8f72f1301db3417db86cf5c0128b0ab4 (table test-table [id=65167cd787274580b02d533613e2e3cd]) successfully deleted
I20260927 18:42:47.181366 27423 ts_tablet_manager.cc:1905] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:47.183249 27025 tablet_service.cc:1467] Processing CreateTablet for tablet c682e834706046c3add56f68bce998f5 (DEFAULT_TABLE table=test-table [id=7a634d585c93443eb5a840c53f443baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:47.183507 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c682e834706046c3add56f68bce998f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:47.183528 27423 ts_tablet_manager.cc:1918] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:47.183534 27164 tablet_service.cc:1467] Processing CreateTablet for tablet c682e834706046c3add56f68bce998f5 (DEFAULT_TABLE table=test-table [id=7a634d585c93443eb5a840c53f443baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:47.183589 27423 log.cc:1196] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/8f72f1301db3417db86cf5c0128b0ab4
I20260927 18:42:47.183773 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c682e834706046c3add56f68bce998f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:47.183791 27423 ts_tablet_manager.cc:1939] T 8f72f1301db3417db86cf5c0128b0ab4 P 36ee887bb35348e08ee022e679098bcd: Deleting consensus metadata
I20260927 18:42:47.185267 26873 catalog_manager.cc:4841] TS 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069): tablet 8f72f1301db3417db86cf5c0128b0ab4 (table test-table [id=65167cd787274580b02d533613e2e3cd]) successfully deleted
I20260927 18:42:47.185678 27401 tablet_bootstrap.cc:492] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5: Bootstrap starting.
I20260927 18:42:47.185984 27399 tablet_bootstrap.cc:492] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Bootstrap starting.
I20260927 18:42:47.186189 27305 tablet_service.cc:1467] Processing CreateTablet for tablet c682e834706046c3add56f68bce998f5 (DEFAULT_TABLE table=test-table [id=7a634d585c93443eb5a840c53f443baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:47.186437 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c682e834706046c3add56f68bce998f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:47.186517 27401 tablet_bootstrap.cc:654] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:47.186761 27399 tablet_bootstrap.cc:654] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:47.187834 27399 tablet_bootstrap.cc:492] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: No bootstrap required, opened a new log
I20260927 18:42:47.187899 27401 tablet_bootstrap.cc:492] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5: No bootstrap required, opened a new log
I20260927 18:42:47.187917 27399 ts_tablet_manager.cc:1397] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:47.187953 27401 ts_tablet_manager.cc:1397] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:47.188280 27401 raft_consensus.cc:348] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.188383 27401 raft_consensus.cc:374] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:47.188414 27401 raft_consensus.cc:729] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7932d91d4334ee6b84eee9cb6d96de5, State: Initialized, Role: FOLLOWER
I20260927 18:42:47.188349 27399 raft_consensus.cc:348] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.188475 27400 tablet_bootstrap.cc:492] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Bootstrap starting.
I20260927 18:42:47.188476 27399 raft_consensus.cc:374] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:47.188517 27399 raft_consensus.cc:729] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Initialized, Role: FOLLOWER
I20260927 18:42:47.188519 27401 consensus_queue.cc:260] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.188642 27399 consensus_queue.cc:260] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.188755 27401 ts_tablet_manager.cc:1428] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:47.188902 27399 ts_tablet_manager.cc:1428] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:47.189266 27400 tablet_bootstrap.cc:654] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:47.190382 27400 tablet_bootstrap.cc:492] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: No bootstrap required, opened a new log
I20260927 18:42:47.190467 27400 ts_tablet_manager.cc:1397] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:47.190788 27400 raft_consensus.cc:348] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.190910 27400 raft_consensus.cc:374] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:47.190951 27400 raft_consensus.cc:729] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ee887bb35348e08ee022e679098bcd, State: Initialized, Role: FOLLOWER
I20260927 18:42:47.191063 27400 consensus_queue.cc:260] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.191283 27400 ts_tablet_manager.cc:1428] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:42:47.503677 27405 raft_consensus.cc:482] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:47.503798 27405 raft_consensus.cc:504] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.504192 27405 leader_election.cc:288] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:47.504591 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c682e834706046c3add56f68bce998f5" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" is_pre_election: true
I20260927 18:42:47.504624 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c682e834706046c3add56f68bce998f5" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ee887bb35348e08ee022e679098bcd" is_pre_election: true
I20260927 18:42:47.504748 27325 raft_consensus.cc:2454] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 0.
I20260927 18:42:47.504766 27045 raft_consensus.cc:2454] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 0.
I20260927 18:42:47.505031 27117 leader_election.cc:302] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7932d91d4334ee6b84eee9cb6d96de5, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:47.505220 27405 raft_consensus.cc:2781] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:47.505295 27405 raft_consensus.cc:482] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:47.505337 27405 raft_consensus.cc:3037] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:47.506242 27405 raft_consensus.cc:504] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.506520 27405 leader_election.cc:288] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 election: Requested vote from peers a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835), 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069)
I20260927 18:42:47.506824 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c682e834706046c3add56f68bce998f5" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7932d91d4334ee6b84eee9cb6d96de5"
I20260927 18:42:47.506944 27045 raft_consensus.cc:3037] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:47.506932 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c682e834706046c3add56f68bce998f5" candidate_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ee887bb35348e08ee022e679098bcd"
I20260927 18:42:47.507064 27325 raft_consensus.cc:3037] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:47.507783 27045 raft_consensus.cc:2454] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 1.
I20260927 18:42:47.508024 27325 raft_consensus.cc:2454] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dfc6f35ca54a8bab6c092dd1859678 in term 1.
I20260927 18:42:47.508036 27117 leader_election.cc:302] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7932d91d4334ee6b84eee9cb6d96de5, e6dfc6f35ca54a8bab6c092dd1859678; no voters:
I20260927 18:42:47.508201 27405 raft_consensus.cc:2781] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:47.508356 27405 raft_consensus.cc:686] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 LEADER]: Becoming Leader. State: Replica: e6dfc6f35ca54a8bab6c092dd1859678, State: Running, Role: LEADER
I20260927 18:42:47.508498 27405 consensus_queue.cc:237] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } }
I20260927 18:42:47.509941 26888 catalog_manager.cc:5495] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130). New cstate: current_term: 1 leader_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dfc6f35ca54a8bab6c092dd1859678" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:47.553013 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:47.550120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.18.235.130:39105)
0927 18:42:47.550322 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:47.550327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:47.550351 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:42:47.550391 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:47.550394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:47.550437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:47.550502 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:47.550723 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:47.551249 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:47.552049 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:47.552279 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:47.552362 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:47.552590 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:47.552661 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:47.552718 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:47.552818 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:42:47.552877 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260927 18:42:47.557929 27325 raft_consensus.cc:1261] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Refusing update from remote peer e6dfc6f35ca54a8bab6c092dd1859678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:47.558017 27045 raft_consensus.cc:1261] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Refusing update from remote peer e6dfc6f35ca54a8bab6c092dd1859678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:47.558290 27478 consensus_queue.cc:1035] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ee887bb35348e08ee022e679098bcd" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:47.558423 27480 consensus_queue.cc:1035] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7932d91d4334ee6b84eee9cb6d96de5" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:47.586215 26888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:47.586340 26888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:47.587936 26888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e9cfed80d5429c860812f5113f3f40: Sending DeleteTablet for 3 replicas of tablet c682e834706046c3add56f68bce998f5
I20260927 18:42:47.588385 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet c682e834706046c3add56f68bce998f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:47 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:49666
I20260927 18:42:47.588434 27305 tablet_service.cc:1514] Processing DeleteTablet for tablet c682e834706046c3add56f68bce998f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:47 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:42564
I20260927 18:42:47.588549 27422 tablet_replica.cc:330] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5: stopping tablet replica
I20260927 18:42:47.588582 27423 tablet_replica.cc:330] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: stopping tablet replica
I20260927 18:42:47.588660 27422 raft_consensus.cc:2229] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:47.588685 27423 raft_consensus.cc:2229] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:47.588760 27422 raft_consensus.cc:2258] T c682e834706046c3add56f68bce998f5 P a7932d91d4334ee6b84eee9cb6d96de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:47.588774 27423 raft_consensus.cc:2258] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:47.588900 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet c682e834706046c3add56f68bce998f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:47 UTC) from {username='oryx', principal='oryx/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:54580
I20260927 18:42:47.589037 27421 tablet_replica.cc:330] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: stopping tablet replica
I20260927 18:42:47.589047 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 26960
I20260927 18:42:47.589151 27421 raft_consensus.cc:2229] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:47.589310 27421 raft_consensus.cc:2258] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:47.590391 27421 ts_tablet_manager.cc:1905] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:47.590979 27423 ts_tablet_manager.cc:1905] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:47.592592 27421 ts_tablet_manager.cc:1918] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:47.592666 27421 log.cc:1196] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/c682e834706046c3add56f68bce998f5
I20260927 18:42:47.592896 27421 ts_tablet_manager.cc:1939] T c682e834706046c3add56f68bce998f5 P e6dfc6f35ca54a8bab6c092dd1859678: Deleting consensus metadata
I20260927 18:42:47.593250 27423 ts_tablet_manager.cc:1918] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:42:47.593321 27423 log.cc:1196] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/c682e834706046c3add56f68bce998f5
I20260927 18:42:47.593545 27423 ts_tablet_manager.cc:1939] T c682e834706046c3add56f68bce998f5 P 36ee887bb35348e08ee022e679098bcd: Deleting consensus metadata
I20260927 18:42:47.593519 26872 catalog_manager.cc:4841] TS e6dfc6f35ca54a8bab6c092dd1859678 (127.18.235.130:39105): tablet c682e834706046c3add56f68bce998f5 (table test-table [id=7a634d585c93443eb5a840c53f443baa]) successfully deleted
I20260927 18:42:47.594161 26873 catalog_manager.cc:4841] TS 36ee887bb35348e08ee022e679098bcd (127.18.235.131:46069): tablet c682e834706046c3add56f68bce998f5 (table test-table [id=7a634d585c93443eb5a840c53f443baa]) successfully deleted
W20260927 18:42:47.595316 26873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:43835 (error 108)
I20260927 18:42:47.595412 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 27100
I20260927 18:42:47.596599 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:47.596294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43835 (local address 127.0.0.1:49680)
0927 18:42:47.596470 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:47.596528 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:43835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
W20260927 18:42:47.596715 26873 catalog_manager.cc:4584] TS a7932d91d4334ee6b84eee9cb6d96de5 (127.18.235.129:43835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c682e834706046c3add56f68bce998f5: Network error: Client connection negotiation failed: client connection to 127.18.235.129:43835: connect: Connection refused (error 111)
I20260927 18:42:47.601603 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 27240
I20260927 18:42:47.606635 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 26856
2026-09-27T18:42:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3204 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:47 dist-test-slave-6lr6 krb5kdc[27491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:47 dist-test-slave-6lr6 krb5kdc[27491](info): set up 2 sockets
Sep 27 18:42:47 dist-test-slave-6lr6 krb5kdc[27491](info): commencing operation
krb5kdc: starting...
W20260927 18:42:48.557678 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:48 dist-test-slave-6lr6 krb5kdc[27491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:48.663257 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:36715
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:48.763396 27504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:48.763599 27504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:48.763634 27504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:48.767477 27504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:48.767519 27504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:48.767536 27504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:48.767554 27504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:48.767570 27504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:48.772339 27504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:36715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:48.773087 27504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:48.773952 27504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:48.781097 27512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:48.781144 27509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:48.781144 27510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:48.781306 27504 server_base.cc:1034] running on GCE node
I20260927 18:42:48.781888 27504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:48.782788 27504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:48.783926 27504 hybrid_clock.cc:648] HybridClock initialized: now 1790534568783904 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T18:42:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1183 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:48 dist-test-slave-6lr6 krb5kdc[27517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:48 dist-test-slave-6lr6 krb5kdc[27517](info): set up 2 sockets
Sep 27 18:42:48 dist-test-slave-6lr6 krb5kdc[27517](info): commencing operation
krb5kdc: starting...
W20260927 18:42:49.741631 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:42:49 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:42:49.844815 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35277
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:49.956503 27530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:49.956734 27530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:49.956782 27530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:49.960691 27530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:49.960745 27530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:49.960775 27530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:49.960860 27530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:49.960899 27530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:49.966071 27530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:49.966882 27530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:49.967722 27530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:49.975232 27536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:49.975234 27538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:49.975262 27535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:49.975363 27530 server_base.cc:1034] running on GCE node
I20260927 18:42:49.975973 27530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:49.976925 27530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:49.978072 27530 hybrid_clock.cc:648] HybridClock initialized: now 1790534569978051 us; error 33 us; skew 500 ppm
Sep 27 18:42:49 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534569, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:49.980412 27530 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:49.981189 27530 webserver.cc:460] Webserver started at http://127.18.235.190:45501/ using document root <none> and password file <none>
I20260927 18:42:49.981657 27530 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:49.981719 27530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:49.981916 27530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:49.983678 27530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "daf9dcf2d4514f38910e3d981cf47a22"
format_stamp: "Formatted at 2026-09-27 18:42:49 on dist-test-slave-6lr6"
server_key: "d179962577c14c03be778adfaf118ccb"
server_key_iv: "ad33a180f5f49aee7a72f17f59b6bb5a"
server_key_version: "encryptionkey@0"
I20260927 18:42:49.984171 27530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "daf9dcf2d4514f38910e3d981cf47a22"
format_stamp: "Formatted at 2026-09-27 18:42:49 on dist-test-slave-6lr6"
server_key: "d179962577c14c03be778adfaf118ccb"
server_key_iv: "ad33a180f5f49aee7a72f17f59b6bb5a"
server_key_version: "encryptionkey@0"
I20260927 18:42:49.987394 27530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 18:42:49.989493 27545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:49.990536 27530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:49.990643 27530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "daf9dcf2d4514f38910e3d981cf47a22"
format_stamp: "Formatted at 2026-09-27 18:42:49 on dist-test-slave-6lr6"
server_key: "d179962577c14c03be778adfaf118ccb"
server_key_iv: "ad33a180f5f49aee7a72f17f59b6bb5a"
server_key_version: "encryptionkey@0"
I20260927 18:42:49.990739 27530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:50.002493 27530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:50.005155 27530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:50.005309 27530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:50.011174 27530 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:35277
I20260927 18:42:50.011184 27597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:35277 every 8 connection(s)
I20260927 18:42:50.011987 27530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:50.014468 27598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:50.019418 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 27530
I20260927 18:42:50.019415 27598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Bootstrap starting.
I20260927 18:42:50.019546 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:42:50.019779 19374 external_mini_cluster.cc:1428] Setting key fb53bc0f5deb6629945da0f5853ba6e1
I20260927 18:42:50.021783 27598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:50.022600 27598 log.cc:824] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:50.024535 27598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: No bootstrap required, opened a new log
I20260927 18:42:50.028151 27598 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.028400 27598 raft_consensus.cc:374] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:50.028465 27598 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf9dcf2d4514f38910e3d981cf47a22, State: Initialized, Role: FOLLOWER
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:50.028887 27598 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.029018 27598 raft_consensus.cc:388] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:50.029083 27598 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:50.029196 27598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:50.030167 27598 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.030506 27598 leader_election.cc:302] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daf9dcf2d4514f38910e3d981cf47a22; no voters:
I20260927 18:42:50.030822 27598 leader_election.cc:288] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:50.030920 27603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:50.031109 27603 raft_consensus.cc:686] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 LEADER]: Becoming Leader. State: Replica: daf9dcf2d4514f38910e3d981cf47a22, State: Running, Role: LEADER
I20260927 18:42:50.031445 27603 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.031983 27598 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:50.032883 27605 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf9dcf2d4514f38910e3d981cf47a22. Latest consensus state: current_term: 1 leader_uuid: "daf9dcf2d4514f38910e3d981cf47a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } } }
I20260927 18:42:50.032995 27605 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:50.033340 27604 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daf9dcf2d4514f38910e3d981cf47a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } } }
I20260927 18:42:50.033427 27604 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:50.034786 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.020936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.18.235.190:35277)
0927 18:42:50.021317 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:50.021334 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:50.021383 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:42:50.022147 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:50.022168 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:50.022558 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:50.022895 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:50.023614 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.024994 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.025886 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.026566 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.026724 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.029064 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:50.029071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:50.029085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:50.029138 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:50.030938 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.031423 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.031428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.031433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.031590 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.031880 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.031883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.031885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.032346 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:50.032528 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:50.033274 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 18:42:50.033579 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
W20260927 18:42:50.036615 27619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:42:50.036757 27619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:42:50.036839 27612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:50.037528 27612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:50.042289 27612 catalog_manager.cc:1347] Generated new cluster ID: a9d8b147bf8644f9b1aa145a6bcf4652
I20260927 18:42:50.042338 27612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:50.054795 27612 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:50.055974 27612 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:50.062860 27612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Generated new TSK 0
I20260927 18:42:50.063447 27612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:42:50.089983 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:35277
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:50.208560 27624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:50.208777 27624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:50.208814 27624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:50.212651 27624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:50.212713 27624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:50.212790 27624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:50.217980 27624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:50.218850 27624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:50.219746 27624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:50.228229 27632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.228214 27630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.228305 27629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:50.228495 27624 server_base.cc:1034] running on GCE node
I20260927 18:42:50.228822 27624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:50.229418 27624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:50.230563 27624 hybrid_clock.cc:648] HybridClock initialized: now 1790534570230546 us; error 30 us; skew 500 ppm
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:50.233155 27624 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:42:50.233925 27624 webserver.cc:460] Webserver started at http://127.18.235.129:44095/ using document root <none> and password file <none>
I20260927 18:42:50.234377 27624 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:50.234431 27624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:50.234581 27624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:50.236419 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "97dbe91b2ffa471dae7e6c9ef4517c4b"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "00b990716baea2229520bb32d94e5ad8"
server_key_iv: "509130489888bcfadf1b17d65906a60a"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.236904 27624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "97dbe91b2ffa471dae7e6c9ef4517c4b"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "00b990716baea2229520bb32d94e5ad8"
server_key_iv: "509130489888bcfadf1b17d65906a60a"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.240119 27624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:42:50.242229 27639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.243281 27624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:42:50.243377 27624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "97dbe91b2ffa471dae7e6c9ef4517c4b"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "00b990716baea2229520bb32d94e5ad8"
server_key_iv: "509130489888bcfadf1b17d65906a60a"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.243458 27624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:50.273989 27624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:50.276569 27624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:50.276724 27624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:50.277276 27624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:50.278019 27624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:50.278064 27624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.278112 27624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:50.278141 27624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.286103 27624 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:44601
I20260927 18:42:50.286128 27752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:44601 every 8 connection(s)
I20260927 18:42:50.286862 27624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:50.295513 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 27624
I20260927 18:42:50.295614 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:50.295863 19374 external_mini_cluster.cc:1428] Setting key 2a93ba5b41848808bf0a9118f36470f2
I20260927 18:42:50.298657 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.288410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:53439 (local address 127.18.235.190:35277)
0927 18:42:50.288516 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:50.288519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:50.289454 (+ 935us) server_negotiation.cc:407] Connection header received
0927 18:42:50.290226 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:50.290230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:50.290283 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:50.290359 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:50.291382 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.291888 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.292883 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.293093 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.293209 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.295304 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:50.295307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:50.295309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:50.295331 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:50.296839 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.297321 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.297324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.297326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.297423 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.297701 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.297704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.297706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.297910 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:50.298003 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:50.298390 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 18:42:50.298504 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:42:50.299429 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.288703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:53439)
0927 18:42:50.289316 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:50.289352 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:50.290047 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:50.290503 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:50.290509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:50.290834 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:50.291248 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:50.291261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.291975 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.291977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:50.292797 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:50.292805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.293232 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.293234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:50.293361 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.293827 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:50.293843 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:50.295149 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:50.296942 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:50.296946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:50.296956 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:50.297218 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:50.297506 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:50.297508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:50.297509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:50.297626 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:50.297991 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:50.297996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:50.298179 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 18:42:50.298326 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:42:50.298546 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":118,"threads_started":1}
I20260927 18:42:50.300446 27753 heartbeater.cc:347] Connected to a master server at 127.18.235.190:35277
I20260927 18:42:50.300633 27753 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:50.301030 27753 heartbeater.cc:511] Master 127.18.235.190:35277 requested a full tablet report, sending...
I20260927 18:42:50.302534 27562 ts_manager.cc:194] Registered new tserver with Master: 97dbe91b2ffa471dae7e6c9ef4517c4b (127.18.235.129:44601)
I20260927 18:42:50.304010 27562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:53439
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:42:50.319353 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:35277
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:50.421082 27758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:50.421298 27758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:50.421340 27758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:50.425215 27758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:50.425256 27758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:50.425318 27758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:50.430137 27758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:50.430964 27758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:50.431823 27758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:50.440311 27766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.440377 27763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.440330 27764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:50.440560 27758 server_base.cc:1034] running on GCE node
I20260927 18:42:50.440958 27758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:50.441512 27758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:50.442658 27758 hybrid_clock.cc:648] HybridClock initialized: now 1790534570442630 us; error 38 us; skew 500 ppm
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:50.445004 27758 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:42:50.445766 27758 webserver.cc:460] Webserver started at http://127.18.235.130:34023/ using document root <none> and password file <none>
I20260927 18:42:50.446271 27758 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:50.446332 27758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:50.446521 27758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:50.448473 27758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "bcd85df69fd6461395e13f18733aefa0"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "061d95dc64c796bebf651c261b640f65"
server_key_iv: "ac6497a4dcb55286c58630e219bf44b2"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.448973 27758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "bcd85df69fd6461395e13f18733aefa0"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "061d95dc64c796bebf651c261b640f65"
server_key_iv: "ac6497a4dcb55286c58630e219bf44b2"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.452224 27758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:42:50.454420 27773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.455470 27758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 18:42:50.455566 27758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "bcd85df69fd6461395e13f18733aefa0"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "061d95dc64c796bebf651c261b640f65"
server_key_iv: "ac6497a4dcb55286c58630e219bf44b2"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.455650 27758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:50.466917 27758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:50.469174 27758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:50.469321 27758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:50.469802 27758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:50.470628 27758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:50.470679 27758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.470731 27758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:50.470785 27758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.478525 27758 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:39499
I20260927 18:42:50.478574 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:39499 every 8 connection(s)
I20260927 18:42:50.479259 27758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:50.483942 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 27758
I20260927 18:42:50.484045 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:50.484326 19374 external_mini_cluster.cc:1428] Setting key 2c37bff64eedbc94954f360c314e254f
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:50.491616 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.480726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:33003 (local address 127.18.235.190:35277)
0927 18:42:50.480888 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:50.480894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:50.481783 (+ 889us) server_negotiation.cc:407] Connection header received
0927 18:42:50.482503 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:50.482505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:50.482549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:50.482615 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:50.483575 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.484160 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.485477 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.485742 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.485824 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.488404 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:50.488407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:50.488409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:50.488438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:50.489819 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.490275 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.490279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.490281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.490367 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.490639 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.490643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.490644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.490862 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:50.490965 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:50.491344 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 18:42:50.491462 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:42:50.492219 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.481049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:33003)
0927 18:42:50.481649 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:50.481684 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:50.482351 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:50.482760 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:50.482767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:50.483066 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:50.483471 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:50.483482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.484277 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.484280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:50.485372 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:50.485382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.485878 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.485882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:50.486063 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.486550 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:50.486568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:50.488252 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:50.489930 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:50.489934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:50.489944 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:50.490170 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:50.490444 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:50.490446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:50.490447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:50.490559 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:50.490948 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:50.490953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:50.491138 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 18:42:50.491273 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 18:42:50.491484 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":104,"threads_started":1}
I20260927 18:42:50.493254 27887 heartbeater.cc:347] Connected to a master server at 127.18.235.190:35277
I20260927 18:42:50.493474 27887 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:42:50.493999 27887 heartbeater.cc:511] Master 127.18.235.190:35277 requested a full tablet report, sending...
I20260927 18:42:50.494889 27562 ts_manager.cc:194] Registered new tserver with Master: bcd85df69fd6461395e13f18733aefa0 (127.18.235.130:39499)
I20260927 18:42:50.495417 27562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:33003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:42:50.507560 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:35277
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:50.608029 27892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:50.608243 27892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:50.608296 27892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:50.612164 27892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:50.612212 27892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:50.612288 27892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:50.617164 27892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:50.618005 27892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:50.618881 27892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:50.627758 27897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.627748 27898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.627758 27900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:50.627982 27892 server_base.cc:1034] running on GCE node
I20260927 18:42:50.628396 27892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:50.628942 27892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:50.630088 27892 hybrid_clock.cc:648] HybridClock initialized: now 1790534570630070 us; error 49 us; skew 500 ppm
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:50.632498 27892 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:42:50.633250 27892 webserver.cc:460] Webserver started at http://127.18.235.131:35679/ using document root <none> and password file <none>
I20260927 18:42:50.633745 27892 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:50.633807 27892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:50.633975 27892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:50.635845 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "3150c44fdaa54cf4a5cc6e35484441a6"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "2e839d419cf124e385e1d80259f97c21"
server_key_iv: "c9abd34d051e5374d0a8e098127ae286"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.636343 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "3150c44fdaa54cf4a5cc6e35484441a6"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "2e839d419cf124e385e1d80259f97c21"
server_key_iv: "c9abd34d051e5374d0a8e098127ae286"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.639663 27892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:42:50.641812 27907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.642876 27892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:42:50.642971 27892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "3150c44fdaa54cf4a5cc6e35484441a6"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "2e839d419cf124e385e1d80259f97c21"
server_key_iv: "c9abd34d051e5374d0a8e098127ae286"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.643054 27892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:50.657974 27892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:50.660490 27892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:50.660638 27892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:50.661151 27892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:50.661868 27892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:50.661926 27892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.661972 27892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:50.661993 27892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.669655 27892 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:42001
I20260927 18:42:50.669708 28020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:42001 every 8 connection(s)
I20260927 18:42:50.670429 27892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:50.671830 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 27892
I20260927 18:42:50.671928 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:50.672181 19374 external_mini_cluster.cc:1428] Setting key 04a9b76bb6db0ec9afcbf22873d3560b
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:42:50.681718 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.671982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:45537 (local address 127.18.235.190:35277)
0927 18:42:50.672122 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:50.672127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:50.673119 (+ 992us) server_negotiation.cc:407] Connection header received
0927 18:42:50.674085 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:50.674088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:50.674138 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:50.674213 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:50.675197 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.675708 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.676726 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.676931 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.676995 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.679017 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:50.679020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:50.679022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:50.679045 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:50.680242 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.680598 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.680600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.680601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.680665 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:50.680895 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:50.680897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:50.680898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:50.681042 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:50.681131 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:50.681522 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:42:50.681609 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:42:50.682379 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:50.672309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:45537)
0927 18:42:50.672972 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:50.673023 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:50.673902 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:50.674367 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:50.674374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:50.674678 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:50.675092 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:50.675104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.675789 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.675792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:50.676641 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:50.676650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:50.677025 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:50.677027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:50.677170 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:50.677607 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:50.677622 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:50.678873 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:50.680318 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:50.680321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:50.680330 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:50.680531 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:50.680723 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:50.680725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:50.680726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:50.680829 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:50.681133 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:50.681139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:50.681345 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 18:42:50.681485 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:42:50.681690 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":127,"threads_started":1}
I20260927 18:42:50.683285 28021 heartbeater.cc:347] Connected to a master server at 127.18.235.190:35277
I20260927 18:42:50.683457 28021 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:50.683842 28021 heartbeater.cc:511] Master 127.18.235.190:35277 requested a full tablet report, sending...
I20260927 18:42:50.684695 27562 ts_manager.cc:194] Registered new tserver with Master: 3150c44fdaa54cf4a5cc6e35484441a6 (127.18.235.131:42001)
I20260927 18:42:50.685261 27562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:45537
I20260927 18:42:50.685667 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab.
Entry for principal oryx/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab.
WARNING: no policy specified for oryx/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab.
Entry for principal oryx/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab.
WARNING: no policy specified for oryx/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab.
Entry for principal oryx/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab.
I20260927 18:42:50.742858 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 27624
I20260927 18:42:50.747412 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 27758
I20260927 18:42:50.752162 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 27892
I20260927 18:42:50.756103 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 27530
I20260927 18:42:50.760665 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35277
--webserver_interface=127.18.235.190
--webserver_port=45501
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:50.861450 28030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:50.861655 28030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:50.861690 28030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:50.865574 28030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:50.865618 28030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:50.865636 28030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:50.865650 28030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:50.865664 28030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:50.870563 28030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=45501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:50.871340 28030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:50.872157 28030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:50.879523 28035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:50.879570 28030 server_base.cc:1034] running on GCE node
W20260927 18:42:50.879606 28036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:50.879515 28038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:50.880342 28030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:50.881301 28030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:50.882452 28030 hybrid_clock.cc:648] HybridClock initialized: now 1790534570882434 us; error 29 us; skew 500 ppm
Sep 27 18:42:50 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534570, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:50.888720 28030 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:42:50.889595 28030 webserver.cc:460] Webserver started at http://127.18.235.190:45501/ using document root <none> and password file <none>
I20260927 18:42:50.890082 28030 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:50.890178 28030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:50.893744 28030 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:50.895659 28045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:50.896741 28030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:50.896858 28030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "daf9dcf2d4514f38910e3d981cf47a22"
format_stamp: "Formatted at 2026-09-27 18:42:49 on dist-test-slave-6lr6"
server_key: "d179962577c14c03be778adfaf118ccb"
server_key_iv: "ad33a180f5f49aee7a72f17f59b6bb5a"
server_key_version: "encryptionkey@0"
I20260927 18:42:50.897361 28030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:50.938937 28030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:50.941813 28030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:50.941972 28030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:50.948331 28030 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:35277
I20260927 18:42:50.948344 28097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:35277 every 8 connection(s)
I20260927 18:42:50.949527 28030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:50.953358 28098 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:42:50.954846 28098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Bootstrap starting.
I20260927 18:42:50.955461 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28030
I20260927 18:42:50.955892 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:44601
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:35277
--webserver_port=44095
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:50.970553 28098 log.cc:824] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:50.976382 28098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:42:50.976670 28098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Bootstrap complete.
I20260927 18:42:50.980450 28098 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.980819 28098 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf9dcf2d4514f38910e3d981cf47a22, State: Initialized, Role: FOLLOWER
I20260927 18:42:50.981266 28098 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.981375 28098 raft_consensus.cc:388] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:50.981432 28098 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:50.981515 28098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:42:50.983191 28098 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.983471 28098 leader_election.cc:302] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daf9dcf2d4514f38910e3d981cf47a22; no voters:
I20260927 18:42:50.983711 28098 leader_election.cc:288] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:42:50.983817 28102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:42:50.984046 28102 raft_consensus.cc:686] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [term 2 LEADER]: Becoming Leader. State: Replica: daf9dcf2d4514f38910e3d981cf47a22, State: Running, Role: LEADER
I20260927 18:42:50.984356 28102 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } }
I20260927 18:42:50.984931 28098 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:50.985716 28103 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "daf9dcf2d4514f38910e3d981cf47a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } } }
I20260927 18:42:50.985841 28103 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:50.986510 28104 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf9dcf2d4514f38910e3d981cf47a22. Latest consensus state: current_term: 2 leader_uuid: "daf9dcf2d4514f38910e3d981cf47a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf9dcf2d4514f38910e3d981cf47a22" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35277 } } }
I20260927 18:42:50.986595 28104 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:50.987015 28112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:50.990350 28112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:50.990952 28112 catalog_manager.cc:1259] Loaded cluster ID: a9d8b147bf8644f9b1aa145a6bcf4652
I20260927 18:42:50.991008 28112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:50.993229 28112 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:50.993850 28112 catalog_manager.cc:5879] T 00000000000000000000000000000000 P daf9dcf2d4514f38910e3d981cf47a22: Loaded TSK: 0
I20260927 18:42:50.994989 28112 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:42:51.061400 28100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:51.061611 28100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:51.061651 28100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:51.065642 28100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:51.065689 28100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:51.065745 28100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:42:51.070586 28100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.129.keytab
--principal=oryx/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:44601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=44095
--tserver_master_addrs=127.18.235.190:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:51.071439 28100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:51.072314 28100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:51.080844 28125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:51.080858 28126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:51.080862 28128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:51.081067 28100 server_base.cc:1034] running on GCE node
I20260927 18:42:51.081529 28100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:51.082064 28100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:51.083222 28100 hybrid_clock.cc:648] HybridClock initialized: now 1790534571083199 us; error 45 us; skew 500 ppm
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534571, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:51.085965 28100 init.cc:377] Logged in from keytab as oryx/127.18.235.129@KRBTEST.COM (short username oryx)
I20260927 18:42:51.086819 28100 webserver.cc:460] Webserver started at http://127.18.235.129:44095/ using document root <none> and password file <none>
I20260927 18:42:51.087316 28100 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:51.087401 28100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:51.090915 28100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:51.092759 28135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.093915 28100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:51.094002 28100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "97dbe91b2ffa471dae7e6c9ef4517c4b"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "00b990716baea2229520bb32d94e5ad8"
server_key_iv: "509130489888bcfadf1b17d65906a60a"
server_key_version: "encryptionkey@0"
I20260927 18:42:51.094499 28100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:51.106870 28100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:51.109550 28100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:51.109697 28100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:51.110193 28100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:51.110929 28100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:51.110980 28100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.111016 28100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:51.111044 28100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.120455 28100 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:44601
I20260927 18:42:51.120505 28248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:44601 every 8 connection(s)
I20260927 18:42:51.121384 28100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.131454 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28100
I20260927 18:42:51.131829 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:39499
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:35277
--webserver_port=34023
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:51.132901 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.123402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:44851)
0927 18:42:51.123978 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.124014 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.124756 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.125833 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.125841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.126205 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.126627 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.126642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.127808 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.127811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.128644 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.128653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.129403 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.129408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.129571 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.130038 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.130053 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.132254 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":103,"threads_started":1}
W20260927 18:42:51.133281 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.123239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:44851 (local address 127.18.235.190:35277)
0927 18:42:51.123626 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.123645 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.124132 (+ 487us) server_negotiation.cc:407] Connection header received
0927 18:42:51.125167 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.125181 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.125465 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.125751 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.126779 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.127687 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.128754 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.129249 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.129389 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.132486 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:44851: BlockingRecv error: recv got EOF from 127.18.235.129:44851 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
W20260927 18:42:51.133486 28249 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:35277 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.235.190:35277: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.138795 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.134041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:56567)
0927 18:42:51.134134 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.134147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.134208 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.134500 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.134502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.134636 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.134741 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.134747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.135660 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.135662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.136195 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.136202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.136759 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.136762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.136828 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.137269 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.137279 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.138708 (+ 1429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:42:51.138960 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.134100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:56567 (local address 127.18.235.190:35277)
0927 18:42:51.134310 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.134313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.134322 (+ 9us) server_negotiation.cc:407] Connection header received
0927 18:42:51.134347 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.134348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.134391 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.134444 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.134844 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.135537 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.136306 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.136643 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.136749 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.138859 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:56567: BlockingRecv error: recv got EOF from 127.18.235.129:56567 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.144160 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.139169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:55807)
0927 18:42:51.139458 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.139472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.139573 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.139768 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.139771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.139950 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.140055 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.140062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.140682 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.140685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.141581 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.141592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.141988 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.141992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.142094 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.142477 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.142490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.144015 (+ 1525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
W20260927 18:42:51.144209 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.139272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:55807 (local address 127.18.235.190:35277)
0927 18:42:51.139371 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.139375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.139543 (+ 168us) server_negotiation.cc:407] Connection header received
0927 18:42:51.139642 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.139644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.139676 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.139736 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.140145 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.140588 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.141682 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.141881 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.141948 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.144129 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:55807: BlockingRecv error: recv got EOF from 127.18.235.129:55807 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 18:42:51.144323 28249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:42:51.236486 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:51.236690 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:51.236727 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:51.240578 28253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:51.240630 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:51.240685 28253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:42:51.245561 28253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.130.keytab
--principal=oryx/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:39499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=34023
--tserver_master_addrs=127.18.235.190:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:51.246409 28253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:51.247327 28253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:51.255823 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:51.255831 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:51.255926 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:51.255987 28253 server_base.cc:1034] running on GCE node
I20260927 18:42:51.256440 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:51.256986 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:51.258122 28253 hybrid_clock.cc:648] HybridClock initialized: now 1790534571258099 us; error 40 us; skew 500 ppm
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534571, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:51.260524 28253 init.cc:377] Logged in from keytab as oryx/127.18.235.130@KRBTEST.COM (short username oryx)
I20260927 18:42:51.261298 28253 webserver.cc:460] Webserver started at http://127.18.235.130:34023/ using document root <none> and password file <none>
I20260927 18:42:51.261771 28253 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:51.261845 28253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:51.265331 28253 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 18:42:51.267107 28269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.268158 28253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:51.268262 28253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "bcd85df69fd6461395e13f18733aefa0"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "061d95dc64c796bebf651c261b640f65"
server_key_iv: "ac6497a4dcb55286c58630e219bf44b2"
server_key_version: "encryptionkey@0"
I20260927 18:42:51.268733 28253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:51.280949 28253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:51.283648 28253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:51.283802 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:51.284296 28253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:51.285086 28253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:51.285146 28253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.285194 28253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:51.285221 28253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.293090 28253 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:39499
I20260927 18:42:51.293139 28382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:39499 every 8 connection(s)
I20260927 18:42:51.293969 28253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:51.298039 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28253
I20260927 18:42:51.298487 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:42001
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:35277
--webserver_port=35679
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:42:51.304018 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.295495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:43869 (local address 127.18.235.190:35277)
0927 18:42:51.295636 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.295641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.296542 (+ 901us) server_negotiation.cc:407] Connection header received
0927 18:42:51.297376 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.297379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.297426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.297486 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.298805 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.299316 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.300545 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.300866 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.300964 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.303892 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:43869: BlockingRecv error: recv got EOF from 127.18.235.130:43869 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:42:51.304445 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.295837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:43869)
0927 18:42:51.296415 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.296451 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.297193 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.297624 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.297631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.298051 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.298671 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.298690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.299434 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.299437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.300431 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.300441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.300988 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.300991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.301182 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.301652 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.301668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.303744 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":105,"threads_started":1}
W20260927 18:42:51.304934 28383 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:35277 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.235.190:35277: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.309476 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.305314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:40795)
0927 18:42:51.305409 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.305421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.305487 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.305677 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.305680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.305808 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.305891 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.305896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.306547 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.306550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.307061 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.307068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.307457 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.307461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.307541 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.307923 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.307933 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.309363 (+ 1430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 18:42:51.309583 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.305406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:40795 (local address 127.18.235.190:35277)
0927 18:42:51.305504 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.305508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.305520 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:51.305554 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.305557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.305593 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.305655 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.305990 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.306455 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.307159 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.307368 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.307450 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.309486 (+ 2036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:40795: BlockingRecv error: recv got EOF from 127.18.235.130:40795 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.314543 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.310001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:59823)
0927 18:42:51.310132 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.310145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.310198 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.310392 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.310393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.310496 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.310553 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.310557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.311471 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.311474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.312017 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.312026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.312521 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.312523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.312573 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.312922 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.312931 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.314439 (+ 1508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 18:42:51.314628 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.310087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:59823 (local address 127.18.235.190:35277)
0927 18:42:51.310188 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.310192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.310204 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:51.310261 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.310264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.310302 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.310376 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.310629 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.311375 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.312107 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.312438 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.312526 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.314539 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:59823: BlockingRecv error: recv got EOF from 127.18.235.130:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 18:42:51.314832 28383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:42:51.401916 28386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:51.402128 28386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:51.402165 28386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:51.405982 28386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:51.406025 28386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:51.406077 28386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:42:51.410851 28386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/krb5kdc/oryx_127.18.235.131.keytab
--principal=oryx/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:42001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=35679
--tserver_master_addrs=127.18.235.190:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:42:51.411666 28386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:51.412513 28386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:51.420925 28391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:51.420909 28394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:51.420900 28392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:51.420991 28386 server_base.cc:1034] running on GCE node
I20260927 18:42:51.421571 28386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:51.422118 28386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:51.423305 28386 hybrid_clock.cc:648] HybridClock initialized: now 1790534571423287 us; error 31 us; skew 500 ppm
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534571, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:51.425808 28386 init.cc:377] Logged in from keytab as oryx/127.18.235.131@KRBTEST.COM (short username oryx)
I20260927 18:42:51.426571 28386 webserver.cc:460] Webserver started at http://127.18.235.131:35679/ using document root <none> and password file <none>
I20260927 18:42:51.427050 28386 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:51.427117 28386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:51.430747 28386 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:42:51.432608 28401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.433786 28386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:42:51.433876 28386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "3150c44fdaa54cf4a5cc6e35484441a6"
format_stamp: "Formatted at 2026-09-27 18:42:50 on dist-test-slave-6lr6"
server_key: "2e839d419cf124e385e1d80259f97c21"
server_key_iv: "c9abd34d051e5374d0a8e098127ae286"
server_key_version: "encryptionkey@0"
I20260927 18:42:51.434360 28386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:51.458973 28386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:51.461391 28386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:51.461552 28386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:51.462036 28386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:51.462764 28386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:51.462807 28386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.462843 28386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:51.462863 28386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:51.471025 28386 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:42001
I20260927 18:42:51.471063 28514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:42001 every 8 connection(s)
I20260927 18:42:51.471956 28386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:51.473817 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28386
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:42:51.483160 28251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.473625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:33909 (local address 127.18.235.190:35277)
0927 18:42:51.473764 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.473769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.475126 (+ 1357us) server_negotiation.cc:407] Connection header received
0927 18:42:51.476015 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.476017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.476061 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.476125 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.477375 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.477996 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.479119 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.479451 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.479549 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.483034 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:33909: BlockingRecv error: recv got EOF from 127.18.235.131:33909 (error 108)
Metrics: {"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":768}
I20260927 18:42:51.483578 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.474739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.18.235.190:35277)
0927 18:42:51.474943 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.474948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.474966 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:42:51.475014 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.475017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.475066 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.475155 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.475903 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.476685 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.477802 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.478023 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.478118 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.479546 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:51.479553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:51.479569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:51.479624 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:51.481379 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:51.481858 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:51.481864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:51.481870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:51.482012 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:51.482310 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:51.482313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:51.482315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:51.482740 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:51.482925 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:51.483208 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 18:42:51.483434 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"spinlock_wait_cycles":25088,"thread_start_us":68,"threads_started":1}
I20260927 18:42:51.483762 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.474053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:33909)
0927 18:42:51.474985 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.475035 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.475819 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.476301 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.476309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.476742 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.477242 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.477264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.478107 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.478111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.478999 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.479009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.479601 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.479605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.479786 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.480330 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.480350 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.482908 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":138,"threads_started":1}
W20260927 18:42:51.484218 28515 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:35277 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.235.190:35277: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.489773 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.485273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:39105)
0927 18:42:51.485553 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.485563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.485634 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.485827 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.485830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.485962 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.486045 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.486050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.486855 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.486858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.487487 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.487496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.487886 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.487889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.487950 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.488257 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.488266 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.489695 (+ 1429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205}
W20260927 18:42:51.489899 28519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.485336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:39105 (local address 127.18.235.190:35277)
0927 18:42:51.485431 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.485434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.485620 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:42:51.485701 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.485703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.485741 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.485801 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.486136 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.486672 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.487578 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.487807 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.487880 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.489804 (+ 1924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:39105: BlockingRecv error: recv got EOF from 127.18.235.131:39105 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:51 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:51.493808 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:51.490161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:33039)
0927 18:42:51.490258 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:51.490285 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:51.490352 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:51.490519 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:51.490521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:51.490625 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:51.490687 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.490691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.491261 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.491263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.491742 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:51.491748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.492063 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.492065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:51.492114 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.492422 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:51.492431 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:51.493734 (+ 1303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:42:51.493916 28519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:51.490232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:33039 (local address 127.18.235.190:35277)
0927 18:42:51.490338 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:51.490341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:51.490353 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:51.490415 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:51.490417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:51.490446 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:51.490502 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:51.490759 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.491185 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.491817 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:51.491999 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:51.492056 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:51.493833 (+ 1777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:33039: BlockingRecv error: recv got EOF from 127.18.235.131:33039 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 18:42:51.494059 28515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:42:52 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:52 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:52.149511 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:52.144815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:49727)
0927 18:42:52.145035 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:52.145048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:52.145149 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:52.145366 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:52.145368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:52.145524 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:52.145634 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:52.145640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.146301 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.146303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:52.146853 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:52.146860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.147217 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.147219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:52.147285 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:52.147694 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:52.147704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:52.149390 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
W20260927 18:42:52.149648 28521 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:52.144873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:49727 (local address 127.18.235.190:35277)
0927 18:42:52.145077 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:52.145080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:52.145092 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:52.145225 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:52.145227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:52.145278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:52.145347 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:52.145714 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.146218 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.146942 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.147146 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.147207 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:52.149523 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:49727: BlockingRecv error: recv got EOF from 127.18.235.129:49727 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
Sep 27 18:42:52 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:52 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:52.320312 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:52.315323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:32859)
0927 18:42:52.315571 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:52.315586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:52.315687 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:52.315914 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:52.315917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:52.316103 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:52.316207 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:52.316213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.316936 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.316940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:52.317535 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:52.317544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.317962 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.317966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:52.318063 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:52.318486 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:52.318496 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:52.320207 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
W20260927 18:42:52.320456 28521 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:52.315442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:32859 (local address 127.18.235.190:35277)
0927 18:42:52.315572 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:52.315577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:52.315638 (+ 61us) server_negotiation.cc:407] Connection header received
0927 18:42:52.315771 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:52.315774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:52.315819 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:52.315881 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:52.316325 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.316838 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.317645 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.317866 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.317948 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:52.320339 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:32859: BlockingRecv error: recv got EOF from 127.18.235.130:32859 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 18:42:52 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:52 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:52.499476 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:52.494540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:38399)
0927 18:42:52.494761 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:52.494773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:52.494863 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:52.495071 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:52.495073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:52.495228 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:52.495341 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:52.495346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.496017 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.496019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:52.496570 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:52.496576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.496927 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.496930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:52.496993 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:52.497495 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:52.497507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:52.499362 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
W20260927 18:42:52.499588 28521 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:52.494603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:38399 (local address 127.18.235.190:35277)
0927 18:42:52.494748 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:52.494752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:52.494830 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:42:52.494943 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:52.494945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:52.494988 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:52.495049 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:52.495430 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.495940 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.496643 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:52.496855 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:52.496919 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:52.499487 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:38399: BlockingRecv error: recv got EOF from 127.18.235.131:38399 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 18:42:53 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:53 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:53.154696 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:53.150099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:52177)
0927 18:42:53.150325 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:53.150338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:53.150416 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:53.150622 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:53.150624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:53.150795 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:53.150891 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:53.150897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.151564 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.151566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:53.152114 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:53.152120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.152481 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.152483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:53.152537 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:53.152978 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:53.152990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:53.154580 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
W20260927 18:42:53.154801 28525 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:53.150173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:52177 (local address 127.18.235.190:35277)
0927 18:42:53.150389 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:53.150392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:53.150404 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:53.150486 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:53.150488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:53.150528 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:53.150599 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:53.150972 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.151490 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.152191 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.152407 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.152472 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:53.154700 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:52177: BlockingRecv error: recv got EOF from 127.18.235.129:52177 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
Sep 27 18:42:53 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:53 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:53.326005 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:53.320964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:47459)
0927 18:42:53.321216 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:53.321230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:53.321321 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:53.321563 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:53.321565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:53.321724 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:53.321816 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:53.321822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.322462 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.322464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:53.323002 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:53.323009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.323538 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.323543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:53.323634 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:53.324180 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:53.324200 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:53.325894 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":64,"threads_started":1}
W20260927 18:42:53.326160 28525 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:53.321030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:47459 (local address 127.18.235.190:35277)
0927 18:42:53.321204 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:53.321208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:53.321290 (+ 82us) server_negotiation.cc:407] Connection header received
0927 18:42:53.321424 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:53.321426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:53.321473 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:53.321541 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:53.321897 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.322387 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.323120 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.323411 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.323529 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:53.326032 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:47459: BlockingRecv error: recv got EOF from 127.18.235.130:47459 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 18:42:53 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:53 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:53.505136 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:53.500101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:56479)
0927 18:42:53.500349 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:53.500361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:53.500452 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:53.500641 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:53.500643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:53.500802 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:53.500893 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:53.500898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.501646 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.501649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:53.502367 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:53.502377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.502909 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.502912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:53.502974 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:53.503429 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:53.503443 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:53.504999 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
W20260927 18:42:53.505244 28525 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:53.500184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:56479 (local address 127.18.235.190:35277)
0927 18:42:53.500299 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:53.500303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:53.500415 (+ 112us) server_negotiation.cc:407] Connection header received
0927 18:42:53.500513 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:53.500516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:53.500556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:53.500617 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:53.500975 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.501548 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.502485 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:53.502803 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:53.502900 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:53.505149 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:56479: BlockingRecv error: recv got EOF from 127.18.235.131:56479 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 18:42:54 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:54 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:54.160673 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:54.155468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:41695)
0927 18:42:54.155721 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:54.155733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:54.155814 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:54.156054 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:54.156056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:54.156217 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:54.156309 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:54.156313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.157278 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.157281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:54.157964 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:54.157974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.158418 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.158421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:54.158477 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:54.158942 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:54.158952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:54.160536 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
W20260927 18:42:54.160780 28529 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:54.155576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:41695 (local address 127.18.235.190:35277)
0927 18:42:54.155853 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:54.155856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:54.155870 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:42:54.155907 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:54.155909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:54.155952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:54.156040 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:54.156398 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.157153 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.158089 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.158323 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.158405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:54.160668 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:41695: BlockingRecv error: recv got EOF from 127.18.235.129:41695 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
Sep 27 18:42:54 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:54 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:54.331468 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:54.326628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:47331)
0927 18:42:54.326870 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:54.326883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:54.326971 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:54.327184 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:54.327186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:54.327346 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:54.327453 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:54.327458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.328123 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.328125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:54.328682 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:54.328689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.329088 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.329091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:54.329171 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:54.329605 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:54.329617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:54.331341 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
W20260927 18:42:54.331558 28529 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:54.326711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:47331 (local address 127.18.235.190:35277)
0927 18:42:54.326849 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:54.326852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:54.326939 (+ 87us) server_negotiation.cc:407] Connection header received
0927 18:42:54.327040 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:54.327042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:54.327088 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:54.327166 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:54.327537 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.328048 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.328776 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.329002 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.329088 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:54.331465 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:47331: BlockingRecv error: recv got EOF from 127.18.235.130:47331 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 18:42:54 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:54 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:54.510568 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:54.505830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:48129)
0927 18:42:54.506094 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:54.506106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:54.506195 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:54.506403 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:54.506404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:54.506558 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:54.506650 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:54.506654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.507354 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.507356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:54.507911 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:54.507918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.508255 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.508257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:54.508311 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:54.508758 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:54.508773 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:54.510444 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
W20260927 18:42:54.510649 28529 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:54.505876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:48129 (local address 127.18.235.190:35277)
0927 18:42:54.506008 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:54.506013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:54.506170 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:42:54.506259 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:54.506262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:54.506311 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:54.506382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:54.506755 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.507273 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.507986 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:54.508185 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:54.508249 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:54.510552 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:48129: BlockingRecv error: recv got EOF from 127.18.235.131:48129 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 18:42:55 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:55 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:55.166085 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:55.161429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:35289)
0927 18:42:55.161673 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:55.161686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:55.161767 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:55.161959 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:55.161961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:55.162130 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:55.162218 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:55.162223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.162904 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.162907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:55.163471 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:55.163479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.163834 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.163836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:55.163891 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:55.164371 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:55.164383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:55.165977 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
W20260927 18:42:55.166226 28533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:55.161487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:35289 (local address 127.18.235.190:35277)
0927 18:42:55.161723 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:55.161727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:55.161739 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:55.161829 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:55.161831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:55.161870 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:55.161938 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:55.162309 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.162824 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.163551 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.163759 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.163823 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:55.166104 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:35289: BlockingRecv error: recv got EOF from 127.18.235.129:35289 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
Sep 27 18:42:55 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:55 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:55.336854 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:55.332151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:41115)
0927 18:42:55.332391 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:55.332403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:55.332497 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:55.332690 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:55.332692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:55.332873 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:55.332969 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:55.332975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.333681 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.333684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:55.334237 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:55.334247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.334602 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.334604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:55.334661 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:55.335135 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:55.335146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:55.336720 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
W20260927 18:42:55.336963 28533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:55.332213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:41115 (local address 127.18.235.190:35277)
0927 18:42:55.332337 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:55.332340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:55.332464 (+ 124us) server_negotiation.cc:407] Connection header received
0927 18:42:55.332562 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:55.332564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:55.332607 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:55.332667 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:55.333051 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.333586 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.334317 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.334528 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.334593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:55.336853 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:41115: BlockingRecv error: recv got EOF from 127.18.235.130:41115 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 18:42:55 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:55 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:55.516177 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:55.511232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:37479)
0927 18:42:55.511464 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:55.511476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:55.511577 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:55.511802 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:55.511804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:55.511989 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:55.512082 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:55.512088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.512802 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.512806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:55.513375 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:55.513391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.513807 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.513810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:55.513889 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:55.514360 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:55.514370 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:55.516039 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
W20260927 18:42:55.516270 28533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:55.511266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:37479 (local address 127.18.235.190:35277)
0927 18:42:55.511396 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:55.511400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:55.511541 (+ 141us) server_negotiation.cc:407] Connection header received
0927 18:42:55.511663 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:55.511666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:55.511709 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:55.511770 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:55.512193 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.512706 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.513497 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:55.513711 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:55.513794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:55.516175 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:37479: BlockingRecv error: recv got EOF from 127.18.235.131:37479 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 18:42:56 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:56 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:56.176012 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.166768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:36629)
0927 18:42:56.167006 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.167019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:56.167096 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:56.167303 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:56.167305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:56.167476 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:56.167568 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:56.167573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.168270 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.168272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:56.168831 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:56.168839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.169230 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.169232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:56.169291 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:56.174217 (+ 4926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:56.174231 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:56.175866 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
W20260927 18:42:56.176115 28537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:56.166846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:36629 (local address 127.18.235.190:35277)
0927 18:42:56.167084 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:56.167088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:56.167100 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:42:56.167170 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:56.167173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:56.167213 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:56.167281 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:56.167659 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.168194 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.168925 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.169156 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.169224 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:56.176004 (+ 6780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:36629: BlockingRecv error: recv got EOF from 127.18.235.129:36629 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
Sep 27 18:42:56 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:56 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:56.342445 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.337513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:45545)
0927 18:42:56.337783 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.337798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:56.337893 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:56.338081 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:56.338083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:56.338281 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:56.338376 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:56.338382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.339094 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.339097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:56.339652 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:56.339659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.340104 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.340108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:56.340181 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:56.340694 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:56.340706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:56.342337 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
W20260927 18:42:56.342571 28537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:56.337634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:45545 (local address 127.18.235.190:35277)
0927 18:42:56.337755 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:56.337759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:56.337856 (+ 97us) server_negotiation.cc:407] Connection header received
0927 18:42:56.337957 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:56.337959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:56.337999 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:56.338059 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:56.338466 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.339016 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.339734 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.339992 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.340074 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:56.342466 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:45545: BlockingRecv error: recv got EOF from 127.18.235.130:45545 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 18:42:56 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:56 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:56.521996 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.516910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:58805)
0927 18:42:56.517184 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.517198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:56.517303 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:56.517541 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:56.517544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:56.517740 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:56.517828 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:56.517834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.518565 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.518568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:56.519151 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:56.519159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.519571 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.519574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:56.519656 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:56.520063 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:56.520073 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:56.521885 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
W20260927 18:42:56.522091 28537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:56.517008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:58805 (local address 127.18.235.190:35277)
0927 18:42:56.517176 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:56.517180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:56.517266 (+ 86us) server_negotiation.cc:407] Connection header received
0927 18:42:56.517385 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:56.517388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:56.517434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:56.517498 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:56.517938 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.518468 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.519262 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:56.519476 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:56.519557 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:56.521999 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:58805: BlockingRecv error: recv got EOF from 127.18.235.131:58805 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 18:42:57 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:57 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:57.181388 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.176668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:38559)
0927 18:42:57.176935 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.176948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:57.177040 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:57.177254 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:57.177256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:57.177427 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:57.177520 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.177525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.178182 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.178183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.178721 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.178728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.179079 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.179081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.179136 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.179618 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:57.179630 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:57.181280 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260927 18:42:57.181516 28541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:57.176734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:38559 (local address 127.18.235.190:35277)
0927 18:42:57.176944 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:57.176948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:57.177006 (+ 58us) server_negotiation.cc:407] Connection header received
0927 18:42:57.177121 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:57.177123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:57.177165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:57.177233 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:57.177600 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.178108 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.178798 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.179007 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.179071 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.181403 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:38559: BlockingRecv error: recv got EOF from 127.18.235.129:38559 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":51,"threads_started":1}
Sep 27 18:42:57 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:57 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:57.347705 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.343102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:38763)
0927 18:42:57.343327 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.343341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:57.343446 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:57.343637 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:57.343639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:57.343809 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:57.343901 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.343907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.344560 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.344562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.345097 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.345131 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.345496 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.345498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.345553 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.345987 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:57.345997 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:57.347593 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
W20260927 18:42:57.347812 28541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:57.343151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:38763 (local address 127.18.235.190:35277)
0927 18:42:57.343285 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:57.343289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:57.343398 (+ 109us) server_negotiation.cc:407] Connection header received
0927 18:42:57.343511 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:57.343514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:57.343554 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:57.343616 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:57.343980 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.344485 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.345202 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.345419 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.345487 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.347712 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:38763: BlockingRecv error: recv got EOF from 127.18.235.130:38763 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 18:42:57 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:57 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:57.527438 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.522877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:34681)
0927 18:42:57.523108 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.523120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:57.523207 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:57.523417 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:57.523419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:57.523579 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:57.523667 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.523672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.524316 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.524318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.524859 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.524865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.525223 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.525225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.525293 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.525742 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:57.525753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:57.527332 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
W20260927 18:42:57.527555 28541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:57.522945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:34681 (local address 127.18.235.190:35277)
0927 18:42:57.523062 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:57.523065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:57.523174 (+ 109us) server_negotiation.cc:407] Connection header received
0927 18:42:57.523287 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:57.523289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:57.523331 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:57.523404 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:57.523744 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.524245 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.524934 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.525153 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.525219 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.527452 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:34681: BlockingRecv error: recv got EOF from 127.18.235.131:34681 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 18:42:58 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:58 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:58.187129 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:58.182004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:51223)
0927 18:42:58.182274 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:58.182287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:58.182379 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:58.182581 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:58.182583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:58.182758 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:58.182852 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:58.182858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.183523 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.183525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:58.184137 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:58.184148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.184700 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.184704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:58.184799 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:58.185295 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:58.185308 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:58.187018 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":64,"threads_started":1}
W20260927 18:42:58.187273 28545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:58.182061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:51223 (local address 127.18.235.190:35277)
0927 18:42:58.182265 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:58.182268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:58.182342 (+ 74us) server_negotiation.cc:407] Connection header received
0927 18:42:58.182446 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:58.182448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:58.182488 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:58.182559 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:58.182930 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.183450 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.184238 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.184567 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.184665 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:58.187143 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:51223: BlockingRecv error: recv got EOF from 127.18.235.129:51223 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
Sep 27 18:42:58 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:58 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:58.353368 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:58.348337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:57849)
0927 18:42:58.348608 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:58.348620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:58.348716 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:58.348916 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:58.348918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:58.349082 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:58.349231 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:58.349240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.350176 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.350180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:58.350743 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:58.350753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.351126 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.351129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:58.351184 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:58.351600 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:58.351610 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:58.353260 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":70,"threads_started":1}
W20260927 18:42:58.353469 28545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:58.348398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:57849 (local address 127.18.235.190:35277)
0927 18:42:58.348547 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:58.348552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:58.348682 (+ 130us) server_negotiation.cc:407] Connection header received
0927 18:42:58.348781 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:58.348783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:58.348823 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:58.348882 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:58.349339 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.350066 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.350839 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.351046 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.351118 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:58.353378 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:57849: BlockingRecv error: recv got EOF from 127.18.235.130:57849 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 18:42:58 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:58 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:58.532648 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:58.528046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:58745)
0927 18:42:58.528325 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:58.528337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:58.528424 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:58.528620 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:58.528622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:58.528785 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:58.528873 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:58.528878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.529574 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.529576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:58.530113 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:58.530118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.530477 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.530479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:58.530535 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:58.530964 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:58.530975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:58.532543 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":63,"threads_started":1}
W20260927 18:42:58.532771 28545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:58.528113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:58745 (local address 127.18.235.190:35277)
0927 18:42:58.528223 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:58.528226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:58.528401 (+ 175us) server_negotiation.cc:407] Connection header received
0927 18:42:58.528495 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:58.528498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:58.528541 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:58.528596 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:58.528950 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.529500 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.530187 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:58.530403 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:58.530466 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:58.532667 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:58745: BlockingRecv error: recv got EOF from 127.18.235.131:58745 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 18:42:59 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:59 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:59.192915 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:59.187730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:47465)
0927 18:42:59.187987 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:59.188000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:59.188083 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:59.188318 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:59.188321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:59.188513 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:59.188613 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:59.188619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.189324 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.189326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:59.189873 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:59.189880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.190277 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.190279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:59.190336 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:59.190777 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:59.190787 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:59.192801 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260927 18:42:59.193030 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:59.187836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:47465 (local address 127.18.235.190:35277)
0927 18:42:59.188043 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:59.188047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:59.188058 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:42:59.188152 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:59.188154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:59.188213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:59.188282 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:59.188712 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.189250 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.189966 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.190192 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.190265 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:59.192919 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:47465: BlockingRecv error: recv got EOF from 127.18.235.129:47465 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
Sep 27 18:42:59 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:59 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:59.359184 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:59.354005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:34857)
0927 18:42:59.354261 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:59.354274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:59.354371 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:59.354575 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:59.354578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:59.354748 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:59.354839 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:59.354845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.355662 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.355663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:59.356195 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:59.356202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.356559 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.356561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:59.356618 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:59.357084 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:59.357098 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:59.359037 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
W20260927 18:42:59.359277 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:59.354082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:34857 (local address 127.18.235.190:35277)
0927 18:42:59.354232 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:59.354237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:59.354333 (+ 96us) server_negotiation.cc:407] Connection header received
0927 18:42:59.354449 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:59.354451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:59.354491 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:59.354552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:59.354924 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.355587 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.356279 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.356485 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.356550 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:59.359168 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:34857: BlockingRecv error: recv got EOF from 127.18.235.130:34857 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 18:42:59 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:42:59 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:42:59.538488 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:59.533321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:48269)
0927 18:42:59.533553 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:59.533568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:59.533661 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:59.533856 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:59.533858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:59.534020 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:59.534111 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:59.534116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.535002 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.535006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:59.535687 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:59.535697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.536085 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.536088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:59.536143 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:59.536608 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:59.536622 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:59.538329 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
W20260927 18:42:59.538556 28549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:42:59.533373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:48269 (local address 127.18.235.190:35277)
0927 18:42:59.533491 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:59.533495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:59.533624 (+ 129us) server_negotiation.cc:407] Connection header received
0927 18:42:59.533725 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:59.533727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:59.533771 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:59.533833 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:59.534187 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.534900 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.535792 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:59.536005 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:59.536078 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:59.538458 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:48269: BlockingRecv error: recv got EOF from 127.18.235.131:48269 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 18:43:00 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:00 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:00.198378 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:00.193551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:47679)
0927 18:43:00.193797 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:00.193810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:00.193895 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:00.194092 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:00.194094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:00.194257 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:00.194349 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:00.194355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.195019 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.195021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:00.195583 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:00.195590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.195944 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.195946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:00.196000 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:00.196464 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:00.196477 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:00.198267 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":65,"threads_started":1}
W20260927 18:43:00.198490 28553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:00.193616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:47679 (local address 127.18.235.190:35277)
0927 18:43:00.193860 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:00.193864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:00.193876 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:43:00.193959 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:00.193961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:00.194001 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:00.194069 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:00.194436 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.194946 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.195661 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.195871 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.195935 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:00.198377 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:47679: BlockingRecv error: recv got EOF from 127.18.235.129:47679 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
Sep 27 18:43:00 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:00 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:00.364581 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:00.359786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:50855)
0927 18:43:00.360031 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:00.360044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:00.360143 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:00.360338 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:00.360340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:00.360522 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:00.360614 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:00.360619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.361299 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.361301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:00.361856 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:00.361865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.362206 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.362209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:00.362265 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:00.362740 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:00.362752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:00.364439 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
W20260927 18:43:00.364667 28553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:00.359874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:50855 (local address 127.18.235.190:35277)
0927 18:43:00.360035 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:00.360039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:00.360093 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:43:00.360211 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:00.360213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:00.360254 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:00.360313 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:00.360692 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.361225 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.361936 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.362137 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.362198 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:00.364571 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:50855: BlockingRecv error: recv got EOF from 127.18.235.130:50855 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 18:43:00 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:00 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:00.544070 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:00.539197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:45985)
0927 18:43:00.539448 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:00.539461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:00.539552 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:00.539755 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:00.539757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:00.539920 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:00.540021 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:00.540026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.540685 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.540687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:00.541263 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:00.541270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.541665 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.541667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:00.541724 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:00.542153 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:00.542164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:00.543929 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
W20260927 18:43:00.544160 28553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:00.539261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:45985 (local address 127.18.235.190:35277)
0927 18:43:00.539388 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:00.539404 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:00.539519 (+ 115us) server_negotiation.cc:407] Connection header received
0927 18:43:00.539619 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:00.539622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:00.539667 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:00.539728 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:00.540102 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.540611 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.541351 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:00.541584 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:00.541658 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:00.544046 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:45985: BlockingRecv error: recv got EOF from 127.18.235.131:45985 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 18:43:01 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:01 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:01.207038 28556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:01.199019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:51869)
0927 18:43:01.199273 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:01.199287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:01.199375 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:01.199578 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:01.199579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:01.199743 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:01.199838 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:01.199843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.200544 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.200546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:01.201136 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:01.201144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.201529 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.201532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:01.201592 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:01.205276 (+ 3684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:01.205288 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:01.206912 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
W20260927 18:43:01.207121 28557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:01.199078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:51869 (local address 127.18.235.190:35277)
0927 18:43:01.199304 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:01.199307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:01.199334 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:43:01.199443 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:01.199446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:01.199486 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:01.199558 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:01.199933 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.200467 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.201219 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.201445 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.201518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:01.207020 (+ 5502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:51869: BlockingRecv error: recv got EOF from 127.18.235.129:51869 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
Sep 27 18:43:01 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:01 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:01.370142 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:01.365321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:43647)
0927 18:43:01.365544 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:01.365557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:01.365646 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:01.365833 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:01.365834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:01.365996 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:01.366086 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:01.366091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.366756 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.366758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:01.367295 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:01.367301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.367644 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.367646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:01.367703 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:01.368177 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:01.368189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:01.370029 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
W20260927 18:43:01.370342 28557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:01.365388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:43647 (local address 127.18.235.190:35277)
0927 18:43:01.365549 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:01.365553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:01.365615 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:43:01.365712 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:01.365714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:01.365758 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:01.365822 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:01.366163 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.366682 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.367369 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.367570 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.367632 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:01.370213 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:43647: BlockingRecv error: recv got EOF from 127.18.235.130:43647 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 18:43:01 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:01 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:01.549580 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:01.544782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:51291)
0927 18:43:01.545034 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:01.545046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:01.545156 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:01.545355 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:01.545357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:01.545529 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:01.545621 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:01.545626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.546340 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.546343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:01.546889 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:01.546897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.547323 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.547327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:01.547405 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:01.547824 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:01.547834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:01.549430 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
W20260927 18:43:01.549654 28557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:01.544898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:51291 (local address 127.18.235.190:35277)
0927 18:43:01.545027 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:01.545030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:01.545115 (+ 85us) server_negotiation.cc:407] Connection header received
0927 18:43:01.545226 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:01.545228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:01.545270 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:01.545331 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:01.545697 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.546229 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.546998 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:01.547221 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:01.547320 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:01.549557 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:51291: BlockingRecv error: recv got EOF from 127.18.235.131:51291 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 18:43:02 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:02 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:02.212512 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:02.207737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:54057)
0927 18:43:02.208000 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:02.208012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:02.208096 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:02.208305 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:02.208306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:02.208468 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:02.208560 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:02.208565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.209246 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.209249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:02.209827 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:02.209835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.210198 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.210201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:02.210267 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:02.210681 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:02.210691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:02.212384 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
W20260927 18:43:02.212651 28561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:02.207812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:54057 (local address 127.18.235.190:35277)
0927 18:43:02.208035 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:02.208038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:02.208054 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:43:02.208172 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:02.208174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:02.208213 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:02.208280 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:02.208642 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.209163 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.209905 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.210113 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.210185 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:02.212522 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:54057: BlockingRecv error: recv got EOF from 127.18.235.129:54057 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
Sep 27 18:43:02 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:02 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:02.375572 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:02.370761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:48353)
0927 18:43:02.370993 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:02.371006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:02.371096 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:02.371282 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:02.371284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:02.371449 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:02.371541 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:02.371547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.372259 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.372263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:02.372863 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:02.372874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.373309 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.373312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:02.373376 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:02.373804 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:02.373815 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:02.375446 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
W20260927 18:43:02.375662 28561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:02.370810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:48353 (local address 127.18.235.190:35277)
0927 18:43:02.370924 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:02.370927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:02.371059 (+ 132us) server_negotiation.cc:407] Connection header received
0927 18:43:02.371155 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:02.371157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:02.371198 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:02.371260 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:02.371639 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.372160 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.372957 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.373215 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.373292 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:02.375569 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:48353: BlockingRecv error: recv got EOF from 127.18.235.130:48353 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 18:43:02 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:02 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:02.554838 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:02.550211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:32929)
0927 18:43:02.550450 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:02.550463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:02.550551 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:02.550745 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:02.550747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:02.550907 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:02.550998 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:02.551003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.551664 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.551667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:02.552209 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:02.552216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.552576 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.552578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:02.552634 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:02.553061 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:02.553071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:02.554728 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
W20260927 18:43:02.554983 28561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:02.550258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:32929 (local address 127.18.235.190:35277)
0927 18:43:02.550382 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:02.550387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:02.550519 (+ 132us) server_negotiation.cc:407] Connection header received
0927 18:43:02.550615 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:02.550618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:02.550663 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:02.550721 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:02.551075 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.551583 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.552296 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:02.552505 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:02.552567 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:02.554862 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:32929: BlockingRecv error: recv got EOF from 127.18.235.131:32929 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 18:43:03 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:03 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:03.218381 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:03.213198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:58299)
0927 18:43:03.213446 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:03.213466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:03.213553 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:03.213928 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:03.213931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:03.214115 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:03.214208 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:03.214214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.214947 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.214952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:03.215514 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:03.215522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.215944 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.215948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:03.216025 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:03.216491 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:03.216503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:03.218240 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
W20260927 18:43:03.218482 28565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:03.213523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:58299 (local address 127.18.235.190:35277)
0927 18:43:03.213725 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:03.213728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:03.213739 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:43:03.213770 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:03.213773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:03.213809 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:03.213886 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:03.214322 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.214840 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.215624 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.215847 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.215939 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:03.218369 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:58299: BlockingRecv error: recv got EOF from 127.18.235.129:58299 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
Sep 27 18:43:03 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:03 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:03.381058 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:03.376269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:44875)
0927 18:43:03.376516 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:03.376535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:03.376631 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:03.376877 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:03.376880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:03.377051 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:03.377193 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:03.377201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.377860 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.377862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:03.378433 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:03.378443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.378805 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.378811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:03.378875 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:03.379295 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:03.379305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:03.380947 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
W20260927 18:43:03.381294 28565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:03.376331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:44875 (local address 127.18.235.190:35277)
0927 18:43:03.376445 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:03.376449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:03.376585 (+ 136us) server_negotiation.cc:407] Connection header received
0927 18:43:03.376704 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:03.376706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:03.376747 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:03.376801 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:03.377287 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.377785 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.378524 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.378729 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.378792 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:03.381073 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:44875: BlockingRecv error: recv got EOF from 127.18.235.130:44875 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 18:43:03 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:03 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:03.560463 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:03.555529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:42441)
0927 18:43:03.555796 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:03.555810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:03.555910 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:03.556129 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:03.556132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:03.556320 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:03.556422 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:03.556429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.557132 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.557135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:03.557699 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:03.557707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.558111 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.558114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:03.558173 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:03.558603 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:03.558615 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:03.560306 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
W20260927 18:43:03.560552 28565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:03.555594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:42441 (local address 127.18.235.190:35277)
0927 18:43:03.555697 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:03.555701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:03.555860 (+ 159us) server_negotiation.cc:407] Connection header received
0927 18:43:03.555981 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:03.555983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:03.556025 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:03.556084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:03.556515 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.557018 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.557795 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:03.558025 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:03.558098 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:03.560455 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:42441: BlockingRecv error: recv got EOF from 127.18.235.131:42441 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 18:43:04 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:04 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:04.223753 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:04.219067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:56033)
0927 18:43:04.219302 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:04.219314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:04.219406 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:04.219610 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:04.219612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:04.219802 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:04.219898 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:04.219904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.220567 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.220569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:04.221167 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:04.221175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.221572 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.221575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:04.221639 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:04.222078 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:04.222089 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:04.223626 (+ 1537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
W20260927 18:43:04.223855 28569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:04.219108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:56033 (local address 127.18.235.190:35277)
0927 18:43:04.219317 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:04.219320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:04.219362 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:43:04.219473 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:04.219476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:04.219517 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:04.219587 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:04.219979 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.220494 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.221250 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.221477 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.221545 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:04.223747 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:56033: BlockingRecv error: recv got EOF from 127.18.235.129:56033 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
Sep 27 18:43:04 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:04 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:04.387120 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:04.381698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:58485)
0927 18:43:04.381960 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:04.381975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:04.382086 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:04.382322 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:04.382325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:04.382527 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:04.382621 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:04.382627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.383342 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.383346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:04.383963 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:04.383974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.384395 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.384399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:04.384480 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:04.384955 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:04.384968 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:04.387006 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
W20260927 18:43:04.387219 28569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:04.381839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:58485 (local address 127.18.235.190:35277)
0927 18:43:04.381960 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:04.381965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:04.382035 (+ 70us) server_negotiation.cc:407] Connection header received
0927 18:43:04.382166 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:04.382169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:04.382222 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:04.382286 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:04.382727 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.383244 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.384071 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.384298 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.384382 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:04.387125 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:58485: BlockingRecv error: recv got EOF from 127.18.235.130:58485 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 18:43:04 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:04 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:04.566587 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:04.561239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:42903)
0927 18:43:04.561533 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:04.561546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:04.561657 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:04.561893 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:04.561895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:04.562095 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:04.562182 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:04.562188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.562889 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.562892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:04.563452 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:04.563459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.563874 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.563877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:04.563958 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:04.564469 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:04.564484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:04.566443 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
W20260927 18:43:04.566660 28569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:04.561302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:42903 (local address 127.18.235.190:35277)
0927 18:43:04.561440 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:04.561445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:04.561611 (+ 166us) server_negotiation.cc:407] Connection header received
0927 18:43:04.561744 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:04.561747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:04.561792 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:04.561855 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:04.562289 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.562791 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.563558 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:04.563777 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:04.563860 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:04.566560 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:42903: BlockingRecv error: recv got EOF from 127.18.235.131:42903 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 18:43:05 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:05 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:05.229339 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:05.224444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:55535)
0927 18:43:05.224685 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:05.224701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:05.224804 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:05.225070 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:05.225073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:05.225282 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:05.225393 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:05.225401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.226131 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.226133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:05.226678 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:05.226684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.227061 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.227063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:05.227120 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:05.227554 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:05.227565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:05.229228 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
W20260927 18:43:05.229498 28573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:05.224470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:55535 (local address 127.18.235.190:35277)
0927 18:43:05.224784 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:05.224789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:05.224806 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:43:05.224881 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:05.224884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:05.224940 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:05.225035 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:05.225513 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.226051 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.226760 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.226981 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.227056 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:05.229364 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:55535: BlockingRecv error: recv got EOF from 127.18.235.129:55535 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":65,"threads_started":1}
Sep 27 18:43:05 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:05 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:05.393067 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:05.387845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:51377)
0927 18:43:05.388099 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:05.388113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:05.388220 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:05.388429 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:05.388431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:05.388612 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:05.388737 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:05.388745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.389462 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.389465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:05.390062 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:05.390073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.390460 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.390462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:05.390526 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:05.391023 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:05.391034 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:05.392955 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
W20260927 18:43:05.393216 28573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:05.387886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:51377 (local address 127.18.235.190:35277)
0927 18:43:05.388015 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:05.388019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:05.388175 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:43:05.388291 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:05.388293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:05.388336 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:05.388410 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:05.388830 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.389356 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.390165 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.390374 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.390446 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:05.393080 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:51377: BlockingRecv error: recv got EOF from 127.18.235.130:51377 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 18:43:05 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:05 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.131@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:05.572378 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:05.567252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.131:37833)
0927 18:43:05.567503 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:05.567516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:05.567619 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:05.567843 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:05.567846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:05.568044 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:05.568132 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:05.568139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.568857 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.568861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:05.569447 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:05.569456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.569864 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.569868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:05.569957 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:05.570382 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:05.570393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:05.572271 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
W20260927 18:43:05.572506 28573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:05.567316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:37833 (local address 127.18.235.190:35277)
0927 18:43:05.567451 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:05.567456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:05.567578 (+ 122us) server_negotiation.cc:407] Connection header received
0927 18:43:05.567700 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:05.567703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:05.567746 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:05.567809 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:05.568241 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.568760 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.569552 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:05.569768 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:05.569851 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:05.572395 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.131:37833: BlockingRecv error: recv got EOF from 127.18.235.131:37833 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 18:43:06 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:06 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.129@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:06.241971 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.230193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.129:51531)
0927 18:43:06.230496 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:06.230510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:06.230614 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:06.230921 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:06.230924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:06.231117 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:06.231216 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:06.231222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.231963 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.231965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:06.232520 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:06.232526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.232886 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.232888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:06.232952 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.240010 (+ 7058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:06.240023 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:06.241803 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":76,"threads_started":1}
W20260927 18:43:06.242194 28577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:06.230252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:51531 (local address 127.18.235.190:35277)
0927 18:43:06.230487 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:06.230491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:06.230563 (+ 72us) server_negotiation.cc:407] Connection header received
0927 18:43:06.230723 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:06.230726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:06.230782 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:06.230873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:06.231305 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.231878 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.232596 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.232812 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.232877 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.241949 (+ 9072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.129:51531: BlockingRecv error: recv got EOF from 127.18.235.129:51531 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":53,"threads_started":1}
Sep 27 18:43:06 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:43:06 dist-test-slave-6lr6 krb5kdc[27517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.235.130@KRBTEST.COM for oryx/127.18.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:43:06.399181 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.394058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35277 (local address 127.18.235.130:33583)
0927 18:43:06.394325 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:06.394337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:06.394429 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:06.394652 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:06.394654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:06.394818 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:06.394922 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:06.394927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.395618 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.395620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:06.396170 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:06.396177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.396613 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.396615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:06.396674 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.397125 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:06.397136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:06.399037 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.235.190:35277: Server oryx/127.18.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
W20260927 18:43:06.399298 28577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:06.394103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:33583 (local address 127.18.235.190:35277)
0927 18:43:06.394238 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:06.394243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:06.394398 (+ 155us) server_negotiation.cc:407] Connection header received
0927 18:43:06.394510 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:06.394512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:06.394566 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:06.394625 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:06.395002 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.395539 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.396279 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.396520 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.396602 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.399180 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.235.130:33583: BlockingRecv error: recv got EOF from 127.18.235.130:33583 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:43:06.477097 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28100
I20260927 18:43:06.481690 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28253
I20260927 18:43:06.486769 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28386
I20260927 18:43:06.490787 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28030
2026-09-27T18:43:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17699 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T18:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:06Z Disabled control of system clock
I20260927 18:43:06.516889 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:46007
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:46007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:43:06.618789 28582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:06.618994 28582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:06.619024 28582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:06.622804 28582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:06.622849 28582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:06.622862 28582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:06.622877 28582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:06.622890 28582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:06.627306 28582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:46007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:06.628086 28582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:06.629223 28582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:06.636713 28587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:06.636735 28590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:06.636763 28588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:06.636880 28582 server_base.cc:1034] running on GCE node
I20260927 18:43:06.637476 28582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:06.638411 28582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:06.639564 28582 hybrid_clock.cc:648] HybridClock initialized: now 1790534586639550 us; error 24 us; skew 500 ppm
I20260927 18:43:06.641258 28582 webserver.cc:460] Webserver started at http://127.18.235.190:42973/ using document root <none> and password file <none>
I20260927 18:43:06.641736 28582 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:06.641798 28582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:06.641975 28582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:06.643756 28582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "8687beb6cce84a8f85d4070729de8c49"
format_stamp: "Formatted at 2026-09-27 18:43:06 on dist-test-slave-6lr6"
server_key: "97f497f5dabdf3299cd4a6ac3aed9d3c"
server_key_iv: "0d76d310c051d55cf2258208468cca71"
server_key_version: "encryptionkey@0"
I20260927 18:43:06.644248 28582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "8687beb6cce84a8f85d4070729de8c49"
format_stamp: "Formatted at 2026-09-27 18:43:06 on dist-test-slave-6lr6"
server_key: "97f497f5dabdf3299cd4a6ac3aed9d3c"
server_key_iv: "0d76d310c051d55cf2258208468cca71"
server_key_version: "encryptionkey@0"
I20260927 18:43:06.647544 28582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260927 18:43:06.649626 28596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:06.650699 28582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:06.650803 28582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8687beb6cce84a8f85d4070729de8c49"
format_stamp: "Formatted at 2026-09-27 18:43:06 on dist-test-slave-6lr6"
server_key: "97f497f5dabdf3299cd4a6ac3aed9d3c"
server_key_iv: "0d76d310c051d55cf2258208468cca71"
server_key_version: "encryptionkey@0"
I20260927 18:43:06.650887 28582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:06.671546 28582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:06.672226 28582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:06.672394 28582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:06.678862 28582 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:46007
I20260927 18:43:06.678874 28648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:46007 every 8 connection(s)
I20260927 18:43:06.679648 28582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:06.681305 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28582
I20260927 18:43:06.681401 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:43:06.681667 19374 external_mini_cluster.cc:1428] Setting key bddebddff097d903b6fe8c8610c7b716
I20260927 18:43:06.682688 28649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:06.688928 28649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49: Bootstrap starting.
I20260927 18:43:06.691323 28649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:06.691356 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.683004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.18.235.190:46007)
0927 18:43:06.683611 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:06.683630 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:06.683671 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:43:06.684477 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:06.684499 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:06.684970 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:06.685341 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:06.685742 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.687116 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.687978 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.688620 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.688768 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.689506 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:06.689513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:06.689527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:06.689988 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:06.690052 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:06.690081 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:06.690174 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:06.690275 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 18:43:06.690515 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260927 18:43:06.692034 28649 log.cc:824] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:06.693734 28649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49: No bootstrap required, opened a new log
I20260927 18:43:06.696820 28649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8687beb6cce84a8f85d4070729de8c49" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46007 } }
I20260927 18:43:06.697012 28649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:06.697068 28649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8687beb6cce84a8f85d4070729de8c49, State: Initialized, Role: FOLLOWER
I20260927 18:43:06.697475 28649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8687beb6cce84a8f85d4070729de8c49" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46007 } }
I20260927 18:43:06.697595 28649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:06.697656 28649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:06.697745 28649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:06.698508 28649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8687beb6cce84a8f85d4070729de8c49" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46007 } }
I20260927 18:43:06.698815 28649 leader_election.cc:302] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8687beb6cce84a8f85d4070729de8c49; no voters:
I20260927 18:43:06.699083 28649 leader_election.cc:288] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:06.699150 28654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:06.699328 28654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [term 1 LEADER]: Becoming Leader. State: Replica: 8687beb6cce84a8f85d4070729de8c49, State: Running, Role: LEADER
I20260927 18:43:06.699622 28654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8687beb6cce84a8f85d4070729de8c49" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46007 } }
I20260927 18:43:06.700119 28649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:06.701077 28656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8687beb6cce84a8f85d4070729de8c49. Latest consensus state: current_term: 1 leader_uuid: "8687beb6cce84a8f85d4070729de8c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8687beb6cce84a8f85d4070729de8c49" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46007 } } }
I20260927 18:43:06.701208 28656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:06.701498 28655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8687beb6cce84a8f85d4070729de8c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8687beb6cce84a8f85d4070729de8c49" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 46007 } } }
I20260927 18:43:06.701645 28655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:06.701560 28663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:06.704033 28663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:06.708575 28663 catalog_manager.cc:1347] Generated new cluster ID: cc549539e07a4a05b5025d5a8f478909
I20260927 18:43:06.708633 28663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:06.724699 28663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:06.725487 28663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:06.737803 28663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8687beb6cce84a8f85d4070729de8c49: Generated new TSK 0
I20260927 18:43:06.738472 28663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:06.748018 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:46007
--builtin_ntp_servers=127.18.235.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:43:06.851239 28673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:06.851461 28673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:06.851508 28673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:06.855515 28673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:06.855558 28673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:06.855628 28673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:43:06.860080 28673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:06.860955 28673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:06.862171 28673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:06.870617 28681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:06.870748 28678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:06.870872 28679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:06.871109 28673 server_base.cc:1034] running on GCE node
I20260927 18:43:06.871446 28673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:06.871984 28673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:06.873164 28673 hybrid_clock.cc:648] HybridClock initialized: now 1790534586873140 us; error 37 us; skew 500 ppm
I20260927 18:43:06.874744 28673 webserver.cc:460] Webserver started at http://127.18.235.129:37333/ using document root <none> and password file <none>
I20260927 18:43:06.875236 28673 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:06.875291 28673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:06.875494 28673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:06.877511 28673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "9c4454ab2d39476782e172dbdb2a2222"
format_stamp: "Formatted at 2026-09-27 18:43:06 on dist-test-slave-6lr6"
server_key: "1abdc780975f767262d32bc40cab9d59"
server_key_iv: "b21b703a1a41af159beba85a84fa66bf"
server_key_version: "encryptionkey@0"
I20260927 18:43:06.878062 28673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "9c4454ab2d39476782e172dbdb2a2222"
format_stamp: "Formatted at 2026-09-27 18:43:06 on dist-test-slave-6lr6"
server_key: "1abdc780975f767262d32bc40cab9d59"
server_key_iv: "b21b703a1a41af159beba85a84fa66bf"
server_key_version: "encryptionkey@0"
I20260927 18:43:06.881333 28673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:43:06.883550 28687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:06.884699 28673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:06.884794 28673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "9c4454ab2d39476782e172dbdb2a2222"
format_stamp: "Formatted at 2026-09-27 18:43:06 on dist-test-slave-6lr6"
server_key: "1abdc780975f767262d32bc40cab9d59"
server_key_iv: "b21b703a1a41af159beba85a84fa66bf"
server_key_version: "encryptionkey@0"
I20260927 18:43:06.884879 28673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:06.899571 28673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:06.900097 28673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:06.900241 28673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:06.900776 28673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:06.901681 28673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:06.901727 28673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:06.901774 28673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:06.901803 28673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:06.909969 28673 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:33531
I20260927 18:43:06.910003 28800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:33531 every 8 connection(s)
I20260927 18:43:06.910732 28673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:06.913460 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28673
I20260927 18:43:06.913545 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:06.913789 19374 external_mini_cluster.cc:1428] Setting key 3097edaabd755c5848f901ee2681b773
I20260927 18:43:06.915645 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:46007
--builtin_ntp_servers=127.18.235.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:43:06.919587 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.912247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:37847 (local address 127.18.235.190:46007)
0927 18:43:06.912356 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:06.912360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:06.913286 (+ 926us) server_negotiation.cc:407] Connection header received
0927 18:43:06.914685 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:06.914689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:06.914745 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:06.914818 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:06.915852 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.916583 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.917834 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.918076 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.918160 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.919143 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:06.919147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:06.919150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:06.919162 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:06.919182 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:06.919189 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:06.919244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:06.919381 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 18:43:06.919452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:43:06.920277 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.912554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:46007 (local address 127.18.235.129:37847)
0927 18:43:06.913148 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:06.913184 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:06.914493 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:06.914976 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:06.914983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:06.915072 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:06.915689 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:06.915711 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.916733 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.916737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:06.917722 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:06.917732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:06.918202 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:06.918206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:06.918348 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:06.918920 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:06.918921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:06.918956 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:06.919012 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:06.919252 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:06.919261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:06.919266 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:43:06.919373 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:43:06.919624 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260927 18:43:06.921461 28801 heartbeater.cc:347] Connected to a master server at 127.18.235.190:46007
I20260927 18:43:06.921644 28801 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:06.922055 28801 heartbeater.cc:511] Master 127.18.235.190:46007 requested a full tablet report, sending...
I20260927 18:43:06.923355 28613 ts_manager.cc:194] Registered new tserver with Master: 9c4454ab2d39476782e172dbdb2a2222 (127.18.235.129:33531)
I20260927 18:43:06.924598 28613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.129:37847
W20260927 18:43:07.017997 28804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:07.018208 28804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:07.018254 28804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:07.022094 28804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:07.022133 28804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:07.022202 28804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:43:07.026613 28804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:07.027446 28804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:07.028594 28804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:07.036891 28812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.037055 28804 server_base.cc:1034] running on GCE node
W20260927 18:43:07.036876 28810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.036868 28809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.037515 28804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:07.038034 28804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:07.039176 28804 hybrid_clock.cc:648] HybridClock initialized: now 1790534587039156 us; error 32 us; skew 500 ppm
I20260927 18:43:07.040697 28804 webserver.cc:460] Webserver started at http://127.18.235.130:33469/ using document root <none> and password file <none>
I20260927 18:43:07.041209 28804 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:07.041270 28804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:07.041455 28804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:07.043236 28804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "38ac77eaada2492a9075f6db5e52dc33"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "fcb3bd3c41e9071399f80cadc20cb118"
server_key_iv: "ed14b13f5b50541757f1d5aaf5e8f14b"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.043725 28804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "38ac77eaada2492a9075f6db5e52dc33"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "fcb3bd3c41e9071399f80cadc20cb118"
server_key_iv: "ed14b13f5b50541757f1d5aaf5e8f14b"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.046911 28804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:43:07.049080 28818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.050190 28804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:07.050282 28804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "38ac77eaada2492a9075f6db5e52dc33"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "fcb3bd3c41e9071399f80cadc20cb118"
server_key_iv: "ed14b13f5b50541757f1d5aaf5e8f14b"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.050377 28804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:07.060949 28804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:07.061498 28804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:07.061633 28804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:07.062100 28804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:07.062888 28804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:07.062934 28804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.062968 28804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:07.062983 28804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.070550 28804 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:46137
I20260927 18:43:07.070573 28931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:46137 every 8 connection(s)
I20260927 18:43:07.071358 28804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:07.079491 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.072978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:58443 (local address 127.18.235.190:46007)
0927 18:43:07.073136 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:07.073141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:07.074096 (+ 955us) server_negotiation.cc:407] Connection header received
0927 18:43:07.075345 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:07.075348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:07.075408 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:07.075488 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:07.076276 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.076775 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.077861 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.078068 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.078154 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:07.079058 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:07.079061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:07.079063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:07.079074 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:07.079092 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:07.079099 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:07.079155 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:07.079281 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:43:07.079342 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:43:07.080175 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.073369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:46007 (local address 127.18.235.130:58443)
0927 18:43:07.073967 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.074003 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:07.075158 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:07.075647 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:07.075654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:07.075729 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:07.076153 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:07.076164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.076883 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.076886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:07.077754 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:07.077763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.078194 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.078198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:07.078337 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:07.078828 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:07.078829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:07.078862 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:07.078929 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:07.079164 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:07.079173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:07.079178 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:43:07.079275 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:43:07.079514 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260927 18:43:07.080711 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28804
I20260927 18:43:07.080813 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:07.081096 19374 external_mini_cluster.cc:1428] Setting key d69997166bc32d39b3d22687e8269b32
I20260927 18:43:07.081255 28932 heartbeater.cc:347] Connected to a master server at 127.18.235.190:46007
I20260927 18:43:07.081468 28932 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:07.081990 28932 heartbeater.cc:511] Master 127.18.235.190:46007 requested a full tablet report, sending...
I20260927 18:43:07.082864 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:46007
--builtin_ntp_servers=127.18.235.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:43:07.082943 28613 ts_manager.cc:194] Registered new tserver with Master: 38ac77eaada2492a9075f6db5e52dc33 (127.18.235.130:46137)
I20260927 18:43:07.083565 28613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.130:58443
W20260927 18:43:07.187788 28935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:07.188090 28935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:07.188163 28935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:07.192612 28935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:07.192742 28935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:07.192847 28935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:43:07.199062 28935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:07.200102 28935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:07.201599 28935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:07.210592 28940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.210592 28943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.210572 28941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.210834 28935 server_base.cc:1034] running on GCE node
I20260927 18:43:07.211264 28935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:07.211859 28935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:07.213019 28935 hybrid_clock.cc:648] HybridClock initialized: now 1790534587212992 us; error 36 us; skew 500 ppm
I20260927 18:43:07.214754 28935 webserver.cc:460] Webserver started at http://127.18.235.131:40345/ using document root <none> and password file <none>
I20260927 18:43:07.215270 28935 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:07.215335 28935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:07.215549 28935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:07.217505 28935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "db9ec9a15f1247f08ff8e4f2ec9f3f6c"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "63da5577708da10d6f4c13753031f434"
server_key_iv: "4417e887d60a7cd8ea2279a2a5afcff5"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.218043 28935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "db9ec9a15f1247f08ff8e4f2ec9f3f6c"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "63da5577708da10d6f4c13753031f434"
server_key_iv: "4417e887d60a7cd8ea2279a2a5afcff5"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.221408 28935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:07.223644 28949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.224704 28935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:07.224813 28935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "db9ec9a15f1247f08ff8e4f2ec9f3f6c"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "63da5577708da10d6f4c13753031f434"
server_key_iv: "4417e887d60a7cd8ea2279a2a5afcff5"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.224910 28935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:07.255939 28935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:07.256507 28935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:07.256664 28935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:07.257210 28935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:07.258065 28935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:07.258116 28935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.258168 28935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:07.258201 28935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.266633 28935 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:33423
I20260927 18:43:07.266654 29062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:33423 every 8 connection(s)
I20260927 18:43:07.267441 28935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:07.268466 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 28935
I20260927 18:43:07.268543 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:07.268774 19374 external_mini_cluster.cc:1428] Setting key 49f07f5d5aa78b274566395f1a1bde1e
I20260927 18:43:07.276032 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.269089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:58813 (local address 127.18.235.190:46007)
0927 18:43:07.269240 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:07.269245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:07.270438 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 18:43:07.271923 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:07.271926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:07.271974 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:07.272055 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:07.272821 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.273403 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.274440 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.274656 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.274757 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:07.275666 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:07.275669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:07.275671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:07.275681 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:07.275720 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:07.275728 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:07.275781 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:07.275884 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:43:07.275930 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:43:07.276793 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.269544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:46007 (local address 127.18.235.131:58813)
0927 18:43:07.270301 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.270354 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:07.271711 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:07.272186 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:07.272191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:07.272271 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:07.272700 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:07.272711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.273489 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.273491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:07.274338 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:07.274347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:07.274803 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:07.274806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:07.274935 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:07.275451 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:07.275453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:07.275489 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:07.275547 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:07.275789 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:07.275794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:07.275798 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:43:07.275880 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:43:07.276083 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":161,"threads_started":1}
I20260927 18:43:07.277711 29063 heartbeater.cc:347] Connected to a master server at 127.18.235.190:46007
I20260927 18:43:07.277917 29063 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:07.278332 29063 heartbeater.cc:511] Master 127.18.235.190:46007 requested a full tablet report, sending...
I20260927 18:43:07.279173 28613 ts_manager.cc:194] Registered new tserver with Master: db9ec9a15f1247f08ff8e4f2ec9f3f6c (127.18.235.131:33423)
I20260927 18:43:07.279675 28613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.131:58813
I20260927 18:43:07.282229 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:43:07.284672 28652 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:07.283840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.18.235.190:46007)
0927 18:43:07.283962 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:07.283966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:07.284096 (+ 130us) server_negotiation.cc:407] Connection header received
0927 18:43:07.284187 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:07.284189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:07.284224 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:07.284289 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:07.284572 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43140: BlockingRecv error: recv got EOF from 127.0.0.1:43140 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:43:07.285049 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28673
I20260927 18:43:07.289749 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28804
I20260927 18:43:07.294245 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28935
I20260927 18:43:07.298586 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 28582
2026-09-27T18:43:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (806 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T18:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:07Z Disabled control of system clock
I20260927 18:43:07.324081 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:43243
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:43243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:43:07.424513 29074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:07.424729 29074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:07.424777 29074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:07.428618 29074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:07.428668 29074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:07.428689 29074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:07.428709 29074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:07.428735 29074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:07.433259 29074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:43243
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:07.434103 29074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:07.435282 29074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:07.442682 29080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.442773 29079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.442682 29082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.442858 29074 server_base.cc:1034] running on GCE node
I20260927 18:43:07.443434 29074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:07.444386 29074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:07.445524 29074 hybrid_clock.cc:648] HybridClock initialized: now 1790534587445506 us; error 32 us; skew 500 ppm
I20260927 18:43:07.447077 29074 webserver.cc:460] Webserver started at http://127.18.235.190:38845/ using document root <none> and password file <none>
I20260927 18:43:07.447556 29074 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:07.447610 29074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:07.447784 29074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:07.449604 29074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "9eb2045d20084205b3c0d39c7c3999ef"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "d7d51e8e1e0995b3693c981407b08048"
server_key_iv: "26736d420a63885e0397289a9f4c6fa2"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.450093 29074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "9eb2045d20084205b3c0d39c7c3999ef"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "d7d51e8e1e0995b3693c981407b08048"
server_key_iv: "26736d420a63885e0397289a9f4c6fa2"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.453296 29074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:43:07.455410 29088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.456475 29074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:07.456575 29074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "9eb2045d20084205b3c0d39c7c3999ef"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "d7d51e8e1e0995b3693c981407b08048"
server_key_iv: "26736d420a63885e0397289a9f4c6fa2"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.456660 29074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:07.460644 29074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:07.461177 29074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:07.461313 29074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:07.467202 29074 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:43243
I20260927 18:43:07.467235 29140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:43243 every 8 connection(s)
I20260927 18:43:07.467959 29074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:07.470796 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:07.475796 29141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef: Bootstrap starting.
I20260927 18:43:07.478053 29141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:07.478281 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29074
I20260927 18:43:07.478389 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:43:07.478674 19374 external_mini_cluster.cc:1428] Setting key fdff34a43423bf994316b23e2d9aaa62
I20260927 18:43:07.478912 29141 log.cc:824] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:07.480834 29141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef: No bootstrap required, opened a new log
I20260927 18:43:07.483274 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.479652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.18.235.190:43243)
0927 18:43:07.480225 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:07.480249 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:07.480281 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:43:07.480847 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:07.480862 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:07.481181 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:07.481474 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:07.482190 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:07.482195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:07.482206 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:07.482216 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:07.482265 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:07.482280 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:07.482369 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:07.482476 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 18:43:07.482683 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260927 18:43:07.484299 29141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb2045d20084205b3c0d39c7c3999ef" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43243 } }
I20260927 18:43:07.484510 29141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:07.484570 29141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb2045d20084205b3c0d39c7c3999ef, State: Initialized, Role: FOLLOWER
I20260927 18:43:07.484974 29141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb2045d20084205b3c0d39c7c3999ef" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43243 } }
I20260927 18:43:07.485152 29141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:07.485239 29141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:07.485342 29141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:07.486137 29141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb2045d20084205b3c0d39c7c3999ef" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43243 } }
I20260927 18:43:07.486454 29141 leader_election.cc:302] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eb2045d20084205b3c0d39c7c3999ef; no voters:
I20260927 18:43:07.486713 29141 leader_election.cc:288] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:07.486781 29146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:07.486971 29146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [term 1 LEADER]: Becoming Leader. State: Replica: 9eb2045d20084205b3c0d39c7c3999ef, State: Running, Role: LEADER
I20260927 18:43:07.487286 29146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb2045d20084205b3c0d39c7c3999ef" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43243 } }
I20260927 18:43:07.487772 29141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:07.488659 29147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eb2045d20084205b3c0d39c7c3999ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb2045d20084205b3c0d39c7c3999ef" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43243 } } }
I20260927 18:43:07.488657 29148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eb2045d20084205b3c0d39c7c3999ef. Latest consensus state: current_term: 1 leader_uuid: "9eb2045d20084205b3c0d39c7c3999ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb2045d20084205b3c0d39c7c3999ef" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43243 } } }
I20260927 18:43:07.488801 29147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:07.488801 29148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:07.489167 29155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:07.491894 29155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:07.496747 29155 catalog_manager.cc:1347] Generated new cluster ID: 805bf8eb28734a55861b1567f8010748
I20260927 18:43:07.496809 29155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:07.528570 29155 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:07.528883 29155 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:07.534752 29155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eb2045d20084205b3c0d39c7c3999ef: Generated new TSK 0
I20260927 18:43:07.535398 29155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:07.544436 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43243
--builtin_ntp_servers=127.18.235.148:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:43:07.647434 29165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:07.647635 29165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:07.647670 29165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:07.651568 29165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:07.651615 29165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:07.651665 29165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:43:07.656080 29165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:07.656872 29165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:07.658061 29165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:07.666527 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.666616 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.666535 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.666657 29165 server_base.cc:1034] running on GCE node
I20260927 18:43:07.667081 29165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:07.667618 29165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:07.668762 29165 hybrid_clock.cc:648] HybridClock initialized: now 1790534587668749 us; error 29 us; skew 500 ppm
I20260927 18:43:07.670377 29165 webserver.cc:460] Webserver started at http://127.18.235.129:34695/ using document root <none> and password file <none>
I20260927 18:43:07.670842 29165 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:07.670895 29165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:07.671046 29165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:07.672816 29165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "2f2034d9bf2846ff8aab15f8f9ee78bf"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "63180b47c3ccfb559f5034abef3bf473"
server_key_iv: "04523b9212cd94f544b85d71dbcb0b5c"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.673322 29165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "2f2034d9bf2846ff8aab15f8f9ee78bf"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "63180b47c3ccfb559f5034abef3bf473"
server_key_iv: "04523b9212cd94f544b85d71dbcb0b5c"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.676924 29165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:43:07.679296 29179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.680342 29165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:07.680430 29165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "2f2034d9bf2846ff8aab15f8f9ee78bf"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "63180b47c3ccfb559f5034abef3bf473"
server_key_iv: "04523b9212cd94f544b85d71dbcb0b5c"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.680524 29165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:07.684556 29165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:07.685047 29165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:07.685215 29165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:07.685668 29165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:07.686440 29165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:07.686482 29165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.686520 29165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:07.686548 29165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.694411 29165 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:32999
I20260927 18:43:07.694448 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:32999 every 8 connection(s)
I20260927 18:43:07.695168 29165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:07.699332 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29165
I20260927 18:43:07.699429 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:07.699668 19374 external_mini_cluster.cc:1428] Setting key 4932216de9e6d17fb57a1e81c511de59
I20260927 18:43:07.700394 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.696727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:42235 (local address 127.18.235.190:43243)
0927 18:43:07.696859 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:07.696862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:07.697815 (+ 953us) server_negotiation.cc:407] Connection header received
0927 18:43:07.698956 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:07.698959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:07.698993 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:07.699060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:07.700004 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:07.700007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:07.700009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:07.700020 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:07.700043 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:07.700051 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:07.700127 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:07.700257 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:43:07.700295 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:43:07.701318 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.697068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43243 (local address 127.18.235.129:42235)
0927 18:43:07.697685 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.697724 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:07.698795 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:07.699174 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:07.699180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:07.699270 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:07.699754 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:07.699757 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:07.699804 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:07.699868 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:07.700128 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:07.700136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:07.700142 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:43:07.700285 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 18:43:07.700496 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260927 18:43:07.701442 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43243
--builtin_ntp_servers=127.18.235.148:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:43:07.702330 29293 heartbeater.cc:347] Connected to a master server at 127.18.235.190:43243
I20260927 18:43:07.702539 29293 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:07.702916 29293 heartbeater.cc:511] Master 127.18.235.190:43243 requested a full tablet report, sending...
I20260927 18:43:07.704239 29105 ts_manager.cc:194] Registered new tserver with Master: 2f2034d9bf2846ff8aab15f8f9ee78bf (127.18.235.129:32999)
W20260927 18:43:07.804864 29296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:07.805089 29296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:07.805163 29296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:07.809036 29296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:07.809090 29296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:07.809193 29296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:43:07.813655 29296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:07.814504 29296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:07.815747 29296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:07.824422 29302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.824410 29301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.824446 29296 server_base.cc:1034] running on GCE node
W20260927 18:43:07.824414 29304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.824961 29296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:07.825538 29296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:07.826754 29296 hybrid_clock.cc:648] HybridClock initialized: now 1790534587826736 us; error 31 us; skew 500 ppm
I20260927 18:43:07.828389 29296 webserver.cc:460] Webserver started at http://127.18.235.130:33515/ using document root <none> and password file <none>
I20260927 18:43:07.828899 29296 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:07.828965 29296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:07.829197 29296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:07.831096 29296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "1cc5664d05164a7cab8cfda794c5e94c"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "14f9c7181a992d574554c627603a65b3"
server_key_iv: "4e6fd207534d14cdc257846dafe6a764"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.831622 29296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "1cc5664d05164a7cab8cfda794c5e94c"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "14f9c7181a992d574554c627603a65b3"
server_key_iv: "4e6fd207534d14cdc257846dafe6a764"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.835047 29296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:43:07.837443 29310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.838551 29296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:43:07.838649 29296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "1cc5664d05164a7cab8cfda794c5e94c"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "14f9c7181a992d574554c627603a65b3"
server_key_iv: "4e6fd207534d14cdc257846dafe6a764"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.838742 29296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:07.843158 29296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:07.843827 29296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:07.843988 29296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:07.844573 29296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:07.845703 29296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:07.845755 29296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.845803 29296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:07.845839 29296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.854640 29296 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42525
I20260927 18:43:07.854671 29423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42525 every 8 connection(s)
I20260927 18:43:07.855412 29296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:07.856676 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29296
I20260927 18:43:07.856789 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:07.857051 19374 external_mini_cluster.cc:1428] Setting key 3ed3ed3230b3077d6f7eec0d4a104f99
I20260927 18:43:07.859045 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43243
--builtin_ntp_servers=127.18.235.148:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:43:07.861377 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.856974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:52235 (local address 127.18.235.190:43243)
0927 18:43:07.857128 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:07.857133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:07.858259 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 18:43:07.859750 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:07.859754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:07.859792 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:07.859858 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:07.860944 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:07.860948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:07.860951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:07.860964 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:07.860987 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:07.860998 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:07.861084 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:07.861206 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:43:07.861249 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:43:07.862084 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.857404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43243 (local address 127.18.235.130:52235)
0927 18:43:07.858103 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.858152 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:07.859531 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:07.860016 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:07.860024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:07.860128 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:07.860700 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:07.860702 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:07.860739 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:07.860802 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:07.861071 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:07.861076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:07.861080 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:43:07.861233 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:43:07.861413 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":132,"threads_started":1}
I20260927 18:43:07.863046 29424 heartbeater.cc:347] Connected to a master server at 127.18.235.190:43243
I20260927 18:43:07.863255 29424 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:07.863646 29424 heartbeater.cc:511] Master 127.18.235.190:43243 requested a full tablet report, sending...
I20260927 18:43:07.864614 29105 ts_manager.cc:194] Registered new tserver with Master: 1cc5664d05164a7cab8cfda794c5e94c (127.18.235.130:42525)
W20260927 18:43:07.962841 29427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:07.963073 29427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:07.963109 29427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:07.967012 29427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:07.967056 29427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:07.967119 29427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:43:07.971612 29427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:07.972445 29427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:07.973726 29427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:07.981915 29433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.981947 29432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:07.981947 29435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:07.982111 29427 server_base.cc:1034] running on GCE node
I20260927 18:43:07.982542 29427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:07.983142 29427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:07.984285 29427 hybrid_clock.cc:648] HybridClock initialized: now 1790534587984261 us; error 34 us; skew 500 ppm
I20260927 18:43:07.985837 29427 webserver.cc:460] Webserver started at http://127.18.235.131:41963/ using document root <none> and password file <none>
I20260927 18:43:07.986322 29427 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:07.986383 29427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:07.986555 29427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:07.988384 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "2353fb1af015405cbb1bcb522074f257"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "0cffa39965c1b37b13a6fe069523df16"
server_key_iv: "fd74902bbe377a88d925887eddeca76e"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.988903 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "2353fb1af015405cbb1bcb522074f257"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "0cffa39965c1b37b13a6fe069523df16"
server_key_iv: "fd74902bbe377a88d925887eddeca76e"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.992141 29427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:43:07.994266 29441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:07.995311 29427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:43:07.995404 29427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "2353fb1af015405cbb1bcb522074f257"
format_stamp: "Formatted at 2026-09-27 18:43:07 on dist-test-slave-6lr6"
server_key: "0cffa39965c1b37b13a6fe069523df16"
server_key_iv: "fd74902bbe377a88d925887eddeca76e"
server_key_version: "encryptionkey@0"
I20260927 18:43:07.995503 29427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:07.999477 29427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:07.999963 29427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:08.000100 29427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:08.000543 29427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:08.001318 29427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:08.001362 29427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:08.001416 29427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:08.001441 29427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:08.008852 29427 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:39257
I20260927 18:43:08.008878 29554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:39257 every 8 connection(s)
I20260927 18:43:08.009634 29427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:08.014724 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:08.011107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:51917 (local address 127.18.235.190:43243)
0927 18:43:08.011229 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:08.011232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:08.012111 (+ 879us) server_negotiation.cc:407] Connection header received
0927 18:43:08.013323 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:08.013326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:08.013369 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:08.013448 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:08.014328 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:08.014332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:08.014335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:08.014346 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:08.014367 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:08.014375 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:08.014455 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:08.014561 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:43:08.014600 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:43:08.015487 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29427
I20260927 18:43:08.015515 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:08.011420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43243 (local address 127.18.235.131:51917)
0927 18:43:08.011986 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:08.012021 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:08.013095 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:08.013598 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:08.013604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:08.013681 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:08.014101 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:08.014103 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:08.014136 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:08.014197 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:08.014440 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:08.014447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:08.014452 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:43:08.014592 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:43:08.014799 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260927 18:43:08.015589 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:08.015874 19374 external_mini_cluster.cc:1428] Setting key 26d589b34feb9951398cd42cbf09f53c
I20260927 18:43:08.016458 29555 heartbeater.cc:347] Connected to a master server at 127.18.235.190:43243
I20260927 18:43:08.016651 29555 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:08.016995 29555 heartbeater.cc:511] Master 127.18.235.190:43243 requested a full tablet report, sending...
I20260927 18:43:08.017835 29105 ts_manager.cc:194] Registered new tserver with Master: 2353fb1af015405cbb1bcb522074f257 (127.18.235.131:39257)
I20260927 18:43:08.028898 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:43:08.031275 29145 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:43:08.030599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.18.235.190:43243)
0927 18:43:08.030709 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:08.030712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:08.030848 (+ 136us) server_negotiation.cc:407] Connection header received
0927 18:43:08.030934 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:08.030936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:08.030960 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:08.031020 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:08.031172 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38288: BlockingRecv error: recv got EOF from 127.0.0.1:38288 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:43:08.031474 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29165
I20260927 18:43:08.036191 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29296
I20260927 18:43:08.040318 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29427
I20260927 18:43:08.044219 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29074
2026-09-27T18:43:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (745 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:08 dist-test-slave-6lr6 krb5kdc[29565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:08 dist-test-slave-6lr6 krb5kdc[29565](info): set up 2 sockets
Sep 27 18:43:08 dist-test-slave-6lr6 krb5kdc[29565](info): commencing operation
krb5kdc: starting...
W20260927 18:43:08.985700 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:43:09.091691 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:37457
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:37457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:09.192269 29578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:09.192526 29578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:09.192582 29578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:09.196525 29578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:09.196575 29578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:09.196601 29578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:09.196622 29578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:09.196651 29578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:09.201689 29578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:37457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:37457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:09.202615 29578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:09.203534 29578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:09.211292 29586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:09.211351 29583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:09.211434 29578 server_base.cc:1034] running on GCE node
W20260927 18:43:09.211310 29584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:09.212074 29578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:09.213035 29578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:09.214188 29578 hybrid_clock.cc:648] HybridClock initialized: now 1790534589214174 us; error 30 us; skew 500 ppm
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:09.216629 29578 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:43:09.217448 29578 webserver.cc:460] Webserver started at http://127.18.235.190:46175/ using document root <none> and password file <none>
I20260927 18:43:09.217934 29578 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:09.217996 29578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:09.218185 29578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:09.220029 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "992b8f844c4f429ba76e54d62adbfd6f"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "c9486dac259906f4ea6b97b2f9491316"
server_key_iv: "b0e3ac89ca437ad93dda31ab6fedd4a8"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.220527 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "992b8f844c4f429ba76e54d62adbfd6f"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "c9486dac259906f4ea6b97b2f9491316"
server_key_iv: "b0e3ac89ca437ad93dda31ab6fedd4a8"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.223860 29578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:43:09.225987 29593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.227073 29578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:09.227174 29578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "992b8f844c4f429ba76e54d62adbfd6f"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "c9486dac259906f4ea6b97b2f9491316"
server_key_iv: "b0e3ac89ca437ad93dda31ab6fedd4a8"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.227257 29578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:09.240756 29578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:09.243408 29578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:09.243558 29578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:09.249748 29578 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:37457
I20260927 18:43:09.250000 29645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:37457 every 8 connection(s)
I20260927 18:43:09.250562 29578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:09.252993 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:09.256081 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29578
I20260927 18:43:09.256182 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:43:09.256404 19374 external_mini_cluster.cc:1428] Setting key e36247860fb32cdec041bd98d363393c
I20260927 18:43:09.258503 29646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: Bootstrap starting.
I20260927 18:43:09.260495 29646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:09.261209 29646 log.cc:824] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:09.262861 29646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: No bootstrap required, opened a new log
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:43:09.265974 29646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992b8f844c4f429ba76e54d62adbfd6f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37457 } }
I20260927 18:43:09.266173 29646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:09.266244 29646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992b8f844c4f429ba76e54d62adbfd6f, State: Initialized, Role: FOLLOWER
I20260927 18:43:09.266644 29646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992b8f844c4f429ba76e54d62adbfd6f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37457 } }
I20260927 18:43:09.266776 29646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:09.266844 29646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:09.266942 29646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:09.267704 29646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992b8f844c4f429ba76e54d62adbfd6f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37457 } }
I20260927 18:43:09.268051 29646 leader_election.cc:302] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 992b8f844c4f429ba76e54d62adbfd6f; no voters:
I20260927 18:43:09.268328 29646 leader_election.cc:288] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:09.268487 29651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:09.268733 29651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [term 1 LEADER]: Becoming Leader. State: Replica: 992b8f844c4f429ba76e54d62adbfd6f, State: Running, Role: LEADER
I20260927 18:43:09.269070 29651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992b8f844c4f429ba76e54d62adbfd6f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37457 } }
I20260927 18:43:09.269529 29646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:09.269651 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.257622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.18.235.190:37457)
0927 18:43:09.258025 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.258039 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.258070 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:43:09.258668 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.258685 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.258978 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.259255 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:09.259990 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.260959 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.261952 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.262440 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.262576 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.264892 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:09.264898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:09.264917 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:09.264941 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:09.266959 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.267327 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.267332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.267338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.267443 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.267685 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.267688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.267690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.268096 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:09.268290 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.268530 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.268806 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260927 18:43:09.271869 29653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "992b8f844c4f429ba76e54d62adbfd6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992b8f844c4f429ba76e54d62adbfd6f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37457 } } }
I20260927 18:43:09.271571 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992b8f844c4f429ba76e54d62adbfd6f. Latest consensus state: current_term: 1 leader_uuid: "992b8f844c4f429ba76e54d62adbfd6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992b8f844c4f429ba76e54d62adbfd6f" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 37457 } } }
I20260927 18:43:09.271982 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:09.271982 29653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f [sys.catalog]: This master's current role is: LEADER
W20260927 18:43:09.274909 29665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:43:09.274978 29665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:43:09.275046 29666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:09.275731 29666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:09.280646 29666 catalog_manager.cc:1347] Generated new cluster ID: ef3515c32729470e8fd93a5b9c1f1b45
I20260927 18:43:09.280696 29666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:09.289778 29666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:09.290596 29666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:09.298406 29666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: Generated new TSK 0
I20260927 18:43:09.298965 29666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:43:09.327672 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37457
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:09.438123 29672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:09.438371 29672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:09.438436 29672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:09.442334 29672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:09.442384 29672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:09.442467 29672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:43:09.447381 29672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:09.448275 29672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:09.449194 29672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:09.457670 29677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:09.457695 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:09.457726 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:09.458069 29672 server_base.cc:1034] running on GCE node
I20260927 18:43:09.458434 29672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:09.458998 29672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:09.460168 29672 hybrid_clock.cc:648] HybridClock initialized: now 1790534589460145 us; error 31 us; skew 500 ppm
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:09.462631 29672 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:43:09.463408 29672 webserver.cc:460] Webserver started at http://127.18.235.129:42013/ using document root <none> and password file <none>
I20260927 18:43:09.463887 29672 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:09.463953 29672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:09.464136 29672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:09.466008 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "5aa4e4b864c642c3b971278d2e10a2fe"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "3c1a37622d32ee5129b35198bffadf74"
server_key_iv: "67220046305e09bb4353f8b9ddb9623b"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.466549 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "5aa4e4b864c642c3b971278d2e10a2fe"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "3c1a37622d32ee5129b35198bffadf74"
server_key_iv: "67220046305e09bb4353f8b9ddb9623b"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.469878 29672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:09.472043 29687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.473088 29672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:09.473238 29672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "5aa4e4b864c642c3b971278d2e10a2fe"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "3c1a37622d32ee5129b35198bffadf74"
server_key_iv: "67220046305e09bb4353f8b9ddb9623b"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.473321 29672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:09.488211 29672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:09.490566 29672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:09.490717 29672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:09.491189 29672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:09.491933 29672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:09.491976 29672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.492023 29672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:09.492051 29672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.500041 29672 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:36339
I20260927 18:43:09.500065 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:36339 every 8 connection(s)
I20260927 18:43:09.500770 29672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:09.502890 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29672
I20260927 18:43:09.502996 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:09.503288 19374 external_mini_cluster.cc:1428] Setting key 16301d480718c47b03997bb295d0f55e
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:43:09.514256 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.502305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:40169 (local address 127.18.235.190:37457)
0927 18:43:09.502417 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.502421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.503192 (+ 771us) server_negotiation.cc:407] Connection header received
0927 18:43:09.504597 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.504600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.504638 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.504699 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:09.506027 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.506560 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.507922 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.508246 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.508345 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.510924 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:09.510928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:09.510931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:09.510958 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:09.512425 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.512866 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.512869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.512870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.512950 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.513258 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.513260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.513261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.513460 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:09.513533 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.514009 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.514105 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:43:09.515000 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.502611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37457 (local address 127.18.235.129:40169)
0927 18:43:09.503050 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.503102 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:09.504398 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:09.504863 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:09.504873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:09.505322 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:09.505905 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.505921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.506677 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.506681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.507796 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.507805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.508363 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.508366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.508502 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.509168 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:09.509190 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:09.510750 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:09.512541 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:09.512546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:09.512556 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:09.512782 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:09.513018 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:09.513020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:09.513022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:09.513173 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:09.513547 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:09.513553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:09.513804 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 18:43:09.513971 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 18:43:09.514251 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
I20260927 18:43:09.516134 29801 heartbeater.cc:347] Connected to a master server at 127.18.235.190:37457
I20260927 18:43:09.516358 29801 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:09.516836 29801 heartbeater.cc:511] Master 127.18.235.190:37457 requested a full tablet report, sending...
I20260927 18:43:09.518311 29610 ts_manager.cc:194] Registered new tserver with Master: 5aa4e4b864c642c3b971278d2e10a2fe (127.18.235.129:36339)
I20260927 18:43:09.519351 29610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:40169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:43:09.527398 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37457
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:09.628113 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:09.628322 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:09.628366 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:09.632300 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:09.632346 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:09.632412 29806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:43:09.637259 29806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:09.638042 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:09.638870 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:09.647097 29814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:09.647143 29812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:09.647122 29811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:09.647285 29806 server_base.cc:1034] running on GCE node
I20260927 18:43:09.647716 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:09.648262 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:09.649437 29806 hybrid_clock.cc:648] HybridClock initialized: now 1790534589649419 us; error 29 us; skew 500 ppm
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:09.651849 29806 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:43:09.652586 29806 webserver.cc:460] Webserver started at http://127.18.235.130:33289/ using document root <none> and password file <none>
I20260927 18:43:09.653048 29806 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:09.653129 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:09.653314 29806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:09.655071 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "af700759928443488cf38b5d3b8c1764"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "49af70bfa3a287d4b5c4820e0bdb6c8d"
server_key_iv: "68431795c93f92b9bfead83ce1d310ef"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.655582 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "af700759928443488cf38b5d3b8c1764"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "49af70bfa3a287d4b5c4820e0bdb6c8d"
server_key_iv: "68431795c93f92b9bfead83ce1d310ef"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.658983 29806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:43:09.661011 29821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.662075 29806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:09.662178 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "af700759928443488cf38b5d3b8c1764"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "49af70bfa3a287d4b5c4820e0bdb6c8d"
server_key_iv: "68431795c93f92b9bfead83ce1d310ef"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.662261 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:09.670893 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:09.673308 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:09.673475 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:09.673944 29806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:09.674619 29806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:09.674664 29806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.674708 29806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:09.674736 29806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.682713 29806 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43511
I20260927 18:43:09.682760 29934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43511 every 8 connection(s)
I20260927 18:43:09.683645 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:43:09.692117 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29806
I20260927 18:43:09.692221 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:09.692476 19374 external_mini_cluster.cc:1428] Setting key 63855a958988adfe9feea82421f146a7
I20260927 18:43:09.695693 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.685453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:50733 (local address 127.18.235.190:37457)
0927 18:43:09.685601 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.685605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.686335 (+ 730us) server_negotiation.cc:407] Connection header received
0927 18:43:09.687555 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.687558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.687601 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.687668 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:09.688650 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.689212 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.690218 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.690419 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.690482 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.692628 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:09.692631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:09.692633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:09.692656 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:09.693915 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.694354 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.694358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.694360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.694430 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.694713 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.694716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.694718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.694885 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:09.694963 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.695404 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.695538 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:43:09.696498 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.685806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37457 (local address 127.18.235.130:50733)
0927 18:43:09.686200 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.686239 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:09.687377 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:09.687786 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:09.687792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:09.688110 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:09.688549 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.688561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.689296 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.689299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.690134 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.690142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.690517 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.690520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.690649 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.691080 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:09.691094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:09.692488 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:09.694013 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:09.694018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:09.694031 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:09.694259 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:09.694506 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:09.694509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:09.694511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:09.694633 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:09.694976 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:09.694982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:09.695191 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 18:43:09.695346 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 18:43:09.695639 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260927 18:43:09.697523 29935 heartbeater.cc:347] Connected to a master server at 127.18.235.190:37457
I20260927 18:43:09.697726 29935 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:09.698133 29935 heartbeater.cc:511] Master 127.18.235.190:37457 requested a full tablet report, sending...
I20260927 18:43:09.699070 29610 ts_manager.cc:194] Registered new tserver with Master: af700759928443488cf38b5d3b8c1764 (127.18.235.130:43511)
I20260927 18:43:09.699592 29610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:50733
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:43:09.715372 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37457
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:09.819205 29940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:09.819428 29940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:09.819479 29940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:09.823362 29940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:09.823414 29940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:09.823508 29940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:43:09.828392 29940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:09.829291 29940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:09.830196 29940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:09.838721 29946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:09.838768 29948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:09.838948 29940 server_base.cc:1034] running on GCE node
W20260927 18:43:09.838778 29945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:09.839422 29940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:09.840022 29940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:09.841162 29940 hybrid_clock.cc:648] HybridClock initialized: now 1790534589841149 us; error 23 us; skew 500 ppm
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:09.847570 29940 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:43:09.848371 29940 webserver.cc:460] Webserver started at http://127.18.235.131:43025/ using document root <none> and password file <none>
I20260927 18:43:09.848860 29940 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:09.848915 29940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:09.849074 29940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:09.851003 29940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "a330b50725ec4da7a33518903d4baed9"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "9a0024d0774c961b72e71fe820b7dcb7"
server_key_iv: "e8b221b3c74b3370724da817c7ce0f12"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.851487 29940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "a330b50725ec4da7a33518903d4baed9"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "9a0024d0774c961b72e71fe820b7dcb7"
server_key_iv: "e8b221b3c74b3370724da817c7ce0f12"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.854950 29940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:43:09.857197 29955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.858307 29940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:43:09.858399 29940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "a330b50725ec4da7a33518903d4baed9"
format_stamp: "Formatted at 2026-09-27 18:43:09 on dist-test-slave-6lr6"
server_key: "9a0024d0774c961b72e71fe820b7dcb7"
server_key_iv: "e8b221b3c74b3370724da817c7ce0f12"
server_key_version: "encryptionkey@0"
I20260927 18:43:09.858489 29940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:09.885375 29940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:09.888082 29940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:09.888226 29940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:09.888721 29940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:09.889468 29940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:09.889513 29940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.889549 29940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:09.889577 29940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:09.897414 29940 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:40647
I20260927 18:43:09.897457 30068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:40647 every 8 connection(s)
I20260927 18:43:09.898181 29940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:09.900063 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 29940
I20260927 18:43:09.900156 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:09.900396 19374 external_mini_cluster.cc:1428] Setting key b02a0efa5d66bc3158cd35c20a9df69d
Sep 27 18:43:09 dist-test-slave-6lr6 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534589, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:43:09.909909 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.899681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:51351 (local address 127.18.235.190:37457)
0927 18:43:09.899800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.899803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.900572 (+ 769us) server_negotiation.cc:407] Connection header received
0927 18:43:09.901946 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.901950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.902003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.902070 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:09.903101 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.903607 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.904661 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.904880 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.904964 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.907038 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:09.907041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:09.907043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:09.907071 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:09.908143 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.908569 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.908573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.908574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.908644 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.908930 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.908934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.908935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.909093 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:09.909214 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.909634 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.909758 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:43:09.910557 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.899991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:37457 (local address 127.18.235.131:51351)
0927 18:43:09.900419 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.900466 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:09.901762 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:09.902246 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:09.902253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:09.902556 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:09.902971 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.902984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.903714 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.903717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.904552 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.904560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.905001 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.905005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.905193 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.905625 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:09.905641 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:09.906896 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:09.908247 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:09.908252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:09.908265 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:09.908481 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:09.908721 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:09.908724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:09.908726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:09.908848 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:09.909219 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:09.909225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:09.909429 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 18:43:09.909579 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:43:09.909851 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260927 18:43:09.911489 30069 heartbeater.cc:347] Connected to a master server at 127.18.235.190:37457
I20260927 18:43:09.911679 30069 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:09.912081 30069 heartbeater.cc:511] Master 127.18.235.190:37457 requested a full tablet report, sending...
I20260927 18:43:09.913060 29609 ts_manager.cc:194] Registered new tserver with Master: a330b50725ec4da7a33518903d4baed9 (127.18.235.131:40647)
I20260927 18:43:09.913707 29609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:51351
I20260927 18:43:09.913812 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:09.921986 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.915717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.18.235.190:37457)
0927 18:43:09.915824 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.915828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.916019 (+ 191us) server_negotiation.cc:407] Connection header received
0927 18:43:09.916116 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.916119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.916157 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.916220 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:09.916823 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.917359 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.918141 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.918343 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.918423 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.919531 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:09.919534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:09.919537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:09.919560 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:09.920530 (+ 970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.920884 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.920887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.920889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.920971 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:09.921233 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:09.921237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:09.921238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:09.921426 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:09.921537 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.921746 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.921842 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:43:09.924361 29609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:09.927106 29609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:09.939802 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.934282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43511 (local address 127.0.0.1:58646)
0927 18:43:09.934569 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.934608 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:09.934710 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:09.935416 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:09.935422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:09.935462 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:09.935584 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.935593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.936969 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.936972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.938301 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.938310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.939331 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.939335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.939441 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.939474 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:43:09.939569 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:43:09.939671 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260927 18:43:09.939988 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.934932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:36339 (local address 127.0.0.1:35426)
0927 18:43:09.935271 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.935284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:09.935361 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:09.936190 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:09.936193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:09.936209 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:09.936295 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.936301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.937454 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.937458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.938561 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.938573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.939553 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.939557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.939661 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.939687 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:43:09.939770 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:43:09.939842 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":44,"thread_start_us":76,"threads_started":1}
I20260927 18:43:09.940356 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.935215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:40647 (local address 127.0.0.1:51550)
0927 18:43:09.935876 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.935887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:09.935966 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:09.936518 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:09.936521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:09.936537 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:09.936628 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.936634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.937895 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.937899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.939022 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:09.939033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.940015 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.940019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:09.940115 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.940136 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:43:09.940181 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 18:43:09.940237 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":27,"thread_start_us":79,"threads_started":1}
I20260927 18:43:09.940364 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.934433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.18.235.130:43511)
0927 18:43:09.934756 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.934766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.934788 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:43:09.934852 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.934856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.935046 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.935207 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:09.935740 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.936836 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.938425 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.939043 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.939150 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.939271 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.940138 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.940234 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260927 18:43:09.940405 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.935131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.18.235.129:36339)
0927 18:43:09.935748 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.935763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.935785 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:43:09.935857 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.935863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.936046 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.936217 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:09.936375 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.937332 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.938728 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.939434 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.939554 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.939678 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.939799 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.940282 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":130,"threads_started":1}
I20260927 18:43:09.940896 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.935534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.18.235.131:40647)
0927 18:43:09.936138 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:09.936148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:09.936167 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:43:09.936223 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:09.936227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:09.936399 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:09.936531 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:09.936722 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.937756 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.939145 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:09.939887 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:09.940540 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:09.940654 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:09.940698 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:43:09.940768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":127,"threads_started":1}
I20260927 18:43:09.942667 29735 tablet_service.cc:1467] Processing CreateTablet for tablet 743bc7d02b6f454992bb6bfd6df7c6cd (DEFAULT_TABLE table=test-table [id=142bbe24a4d84a56be50129966e45090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:09.942714 29869 tablet_service.cc:1467] Processing CreateTablet for tablet 743bc7d02b6f454992bb6bfd6df7c6cd (DEFAULT_TABLE table=test-table [id=142bbe24a4d84a56be50129966e45090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:09.943681 29735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743bc7d02b6f454992bb6bfd6df7c6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:09.943895 29869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743bc7d02b6f454992bb6bfd6df7c6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:09.948045 30003 tablet_service.cc:1467] Processing CreateTablet for tablet 743bc7d02b6f454992bb6bfd6df7c6cd (DEFAULT_TABLE table=test-table [id=142bbe24a4d84a56be50129966e45090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:09.948992 30003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743bc7d02b6f454992bb6bfd6df7c6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:09.951444 30086 tablet_bootstrap.cc:492] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: Bootstrap starting.
I20260927 18:43:09.952999 30087 tablet_bootstrap.cc:492] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Bootstrap starting.
I20260927 18:43:09.954121 30086 tablet_bootstrap.cc:654] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:09.954934 30086 log.cc:824] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:09.955477 30087 tablet_bootstrap.cc:654] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:09.956269 30087 log.cc:824] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:09.956642 30089 tablet_bootstrap.cc:492] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Bootstrap starting.
I20260927 18:43:09.957926 30086 tablet_bootstrap.cc:492] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: No bootstrap required, opened a new log
I20260927 18:43:09.958559 30087 tablet_bootstrap.cc:492] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: No bootstrap required, opened a new log
I20260927 18:43:09.958748 30087 ts_tablet_manager.cc:1397] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:43:09.959455 30089 tablet_bootstrap.cc:654] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:09.958112 30086 ts_tablet_manager.cc:1397] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 18:43:09.960649 30089 log.cc:824] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:09.962615 30087 raft_consensus.cc:348] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:09.962924 30087 raft_consensus.cc:374] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:09.962998 30087 raft_consensus.cc:729] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af700759928443488cf38b5d3b8c1764, State: Initialized, Role: FOLLOWER
I20260927 18:43:09.963466 30087 consensus_queue.cc:260] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:09.963732 30086 raft_consensus.cc:348] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:09.963960 30086 raft_consensus.cc:374] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:09.964001 30086 raft_consensus.cc:729] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa4e4b864c642c3b971278d2e10a2fe, State: Initialized, Role: FOLLOWER
I20260927 18:43:09.964373 30086 consensus_queue.cc:260] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:09.964454 30087 ts_tablet_manager.cc:1428] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:43:09.965068 29801 heartbeater.cc:503] Master 127.18.235.190:37457 was elected leader, sending a full tablet report...
I20260927 18:43:09.965294 30086 ts_tablet_manager.cc:1428] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:43:09.966130 29935 heartbeater.cc:503] Master 127.18.235.190:37457 was elected leader, sending a full tablet report...
I20260927 18:43:09.967543 30089 tablet_bootstrap.cc:492] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: No bootstrap required, opened a new log
I20260927 18:43:09.967713 30089 ts_tablet_manager.cc:1397] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 18:43:09.971372 30089 raft_consensus.cc:348] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:09.971606 30089 raft_consensus.cc:374] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:09.971657 30089 raft_consensus.cc:729] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a330b50725ec4da7a33518903d4baed9, State: Initialized, Role: FOLLOWER
I20260927 18:43:09.972046 30089 consensus_queue.cc:260] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:09.973296 30089 ts_tablet_manager.cc:1428] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:43:09.973647 30069 heartbeater.cc:503] Master 127.18.235.190:37457 was elected leader, sending a full tablet report...
W20260927 18:43:10.001696 29802 tablet.cc:2369] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:10.014284 30094 raft_consensus.cc:482] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:10.014446 30094 raft_consensus.cc:504] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:10.015422 30094 leader_election.cc:288] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af700759928443488cf38b5d3b8c1764 (127.18.235.130:43511), 5aa4e4b864c642c3b971278d2e10a2fe (127.18.235.129:36339)
I20260927 18:43:10.019076 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.015638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43511 (local address 127.18.235.131:37995)
0927 18:43:10.015778 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:10.015790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:10.015873 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:10.016092 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:10.016094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:10.016110 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:10.016210 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:10.016215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.017081 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.017084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:10.017984 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:10.017992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.018694 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.018697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:10.018797 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:10.018822 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:43:10.018890 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:43:10.018958 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:43:10.019083 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.015776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:37995 (local address 127.18.235.130:43511)
0927 18:43:10.015898 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:10.015903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:10.015929 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:43:10.015975 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:10.015977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:10.016016 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:10.016075 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:10.016331 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.016985 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.018098 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.018597 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.018684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:10.018726 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:10.018902 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 18:43:10.018968 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:43:10.019276 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.016191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:56845 (local address 127.18.235.129:36339)
0927 18:43:10.016345 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:10.016349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:10.016359 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:43:10.016398 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:10.016400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:10.016438 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:10.016492 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:10.016757 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.017348 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.018372 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.018791 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.018901 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:10.018953 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:10.019113 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:43:10.019178 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:43:10.019459 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.015875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:36339 (local address 127.18.235.131:56845)
0927 18:43:10.016141 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:10.016153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:10.016239 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:10.016523 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:10.016525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:10.016542 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:10.016646 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:10.016651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.017438 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.017440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:10.018257 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:10.018266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.018895 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.018899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:10.018999 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:10.019021 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:43:10.019091 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:43:10.019270 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260927 18:43:10.019770 29889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743bc7d02b6f454992bb6bfd6df7c6cd" candidate_uuid: "a330b50725ec4da7a33518903d4baed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af700759928443488cf38b5d3b8c1764" is_pre_election: true
I20260927 18:43:10.019938 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743bc7d02b6f454992bb6bfd6df7c6cd" candidate_uuid: "a330b50725ec4da7a33518903d4baed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" is_pre_election: true
I20260927 18:43:10.020035 29889 raft_consensus.cc:2454] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a330b50725ec4da7a33518903d4baed9 in term 0.
I20260927 18:43:10.020190 29755 raft_consensus.cc:2454] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a330b50725ec4da7a33518903d4baed9 in term 0.
I20260927 18:43:10.020439 29957 leader_election.cc:302] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a330b50725ec4da7a33518903d4baed9, af700759928443488cf38b5d3b8c1764; no voters:
I20260927 18:43:10.020650 30094 raft_consensus.cc:2781] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:10.020730 30094 raft_consensus.cc:482] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:10.020776 30094 raft_consensus.cc:3037] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:10.021756 30094 raft_consensus.cc:504] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:10.022035 30094 leader_election.cc:288] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [CANDIDATE]: Term 1 election: Requested vote from peers af700759928443488cf38b5d3b8c1764 (127.18.235.130:43511), 5aa4e4b864c642c3b971278d2e10a2fe (127.18.235.129:36339)
I20260927 18:43:10.022332 29889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743bc7d02b6f454992bb6bfd6df7c6cd" candidate_uuid: "a330b50725ec4da7a33518903d4baed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af700759928443488cf38b5d3b8c1764"
I20260927 18:43:10.022420 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743bc7d02b6f454992bb6bfd6df7c6cd" candidate_uuid: "a330b50725ec4da7a33518903d4baed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4e4b864c642c3b971278d2e10a2fe"
I20260927 18:43:10.022469 29889 raft_consensus.cc:3037] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:10.022511 29755 raft_consensus.cc:3037] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:10.023622 29755 raft_consensus.cc:2454] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a330b50725ec4da7a33518903d4baed9 in term 1.
I20260927 18:43:10.023622 29889 raft_consensus.cc:2454] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a330b50725ec4da7a33518903d4baed9 in term 1.
I20260927 18:43:10.023873 29957 leader_election.cc:302] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a330b50725ec4da7a33518903d4baed9, af700759928443488cf38b5d3b8c1764; no voters:
I20260927 18:43:10.024013 30094 raft_consensus.cc:2781] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:10.024185 30094 raft_consensus.cc:686] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 1 LEADER]: Becoming Leader. State: Replica: a330b50725ec4da7a33518903d4baed9, State: Running, Role: LEADER
I20260927 18:43:10.024470 30094 consensus_queue.cc:237] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } }
I20260927 18:43:10.027249 29609 catalog_manager.cc:5495] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a330b50725ec4da7a33518903d4baed9 (127.18.235.131). New cstate: current_term: 1 leader_uuid: "a330b50725ec4da7a33518903d4baed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a330b50725ec4da7a33518903d4baed9" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40647 } health_report { overall_health: HEALTHY } } }
I20260927 18:43:10.037191 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.034086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.18.235.131:40647)
0927 18:43:10.034191 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:10.034194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:10.034207 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:43:10.034322 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:10.034324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:10.034372 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:10.034452 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:10.034628 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.035089 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.036020 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:10.036222 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:10.036300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:10.036510 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:10.036865 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:10.036953 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:10.036989 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:43:10.037032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:43:10.042110 29755 raft_consensus.cc:1261] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 1 FOLLOWER]: Refusing update from remote peer a330b50725ec4da7a33518903d4baed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:10.042116 29889 raft_consensus.cc:1261] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 1 FOLLOWER]: Refusing update from remote peer a330b50725ec4da7a33518903d4baed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:10.042703 30096 consensus_queue.cc:1035] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af700759928443488cf38b5d3b8c1764" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:10.042858 30094 consensus_queue.cc:1035] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa4e4b864c642c3b971278d2e10a2fe" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:10.051569 30100 mvcc.cc:205] Tried to move back new op lower bound from 7334029680806555648 to 7334029680742731776. Current Snapshot: MvccSnapshot[applied={T|T < 7334029680806555648}]
I20260927 18:43:10.057029 29610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:10.057196 29610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:10.059383 29610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 992b8f844c4f429ba76e54d62adbfd6f: Sending DeleteTablet for 3 replicas of tablet 743bc7d02b6f454992bb6bfd6df7c6cd
I20260927 18:43:10.060025 29869 tablet_service.cc:1514] Processing DeleteTablet for tablet 743bc7d02b6f454992bb6bfd6df7c6cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:10 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:58646
I20260927 18:43:10.060268 29735 tablet_service.cc:1514] Processing DeleteTablet for tablet 743bc7d02b6f454992bb6bfd6df7c6cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:10 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:35426
I20260927 18:43:10.060443 30109 tablet_replica.cc:330] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: stopping tablet replica
I20260927 18:43:10.060683 30110 tablet_replica.cc:330] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe: stopping tablet replica
I20260927 18:43:10.060747 30109 raft_consensus.cc:2229] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:10.060840 30003 tablet_service.cc:1514] Processing DeleteTablet for tablet 743bc7d02b6f454992bb6bfd6df7c6cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:10 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:51550
I20260927 18:43:10.061033 30109 raft_consensus.cc:2258] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:10.061071 30110 raft_consensus.cc:2229] T 743bc7d02b6f454992bb6bfd6df7c6cd P 5aa4e4b864c642c3b971278d2e10a2fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:10.061614 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29672
I20260927 18:43:10.062644 30109 ts_tablet_manager.cc:1905] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:10.065372 30109 ts_tablet_manager.cc:1918] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:10.065474 30109 log.cc:1196] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/743bc7d02b6f454992bb6bfd6df7c6cd
I20260927 18:43:10.065848 30109 ts_tablet_manager.cc:1939] T 743bc7d02b6f454992bb6bfd6df7c6cd P af700759928443488cf38b5d3b8c1764: Deleting consensus metadata
I20260927 18:43:10.066949 29595 catalog_manager.cc:4841] TS af700759928443488cf38b5d3b8c1764 (127.18.235.130:43511): tablet 743bc7d02b6f454992bb6bfd6df7c6cd (table test-table [id=142bbe24a4d84a56be50129966e45090]) successfully deleted
W20260927 18:43:10.069344 29595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:36339 (error 108)
I20260927 18:43:10.069442 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29806
I20260927 18:43:10.069502 30111 tablet_replica.cc:330] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: stopping tablet replica
I20260927 18:43:10.070363 30111 raft_consensus.cc:2229] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:10.070689 30111 raft_consensus.cc:2258] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:10.070732 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.070461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:36339 (local address 127.0.0.1:35442)
0927 18:43:10.070604 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:10.070662 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 18:43:10.070864 29595 catalog_manager.cc:4584] TS 5aa4e4b864c642c3b971278d2e10a2fe (127.18.235.129:36339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 743bc7d02b6f454992bb6bfd6df7c6cd: Network error: Client connection negotiation failed: client connection to 127.18.235.129:36339: connect: Connection refused (error 111)
I20260927 18:43:10.072045 30111 ts_tablet_manager.cc:1905] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:10.074664 30111 ts_tablet_manager.cc:1918] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:10.074777 30111 log.cc:1196] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/743bc7d02b6f454992bb6bfd6df7c6cd
I20260927 18:43:10.075110 30111 ts_tablet_manager.cc:1939] T 743bc7d02b6f454992bb6bfd6df7c6cd P a330b50725ec4da7a33518903d4baed9: Deleting consensus metadata
I20260927 18:43:10.076205 29595 catalog_manager.cc:4841] TS a330b50725ec4da7a33518903d4baed9 (127.18.235.131:40647): tablet 743bc7d02b6f454992bb6bfd6df7c6cd (table test-table [id=142bbe24a4d84a56be50129966e45090]) successfully deleted
I20260927 18:43:10.076710 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29940
I20260927 18:43:10.081447 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 29578
2026-09-27T18:43:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2038 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:10 dist-test-slave-6lr6 krb5kdc[30115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:10 dist-test-slave-6lr6 krb5kdc[30115](info): set up 2 sockets
Sep 27 18:43:10 dist-test-slave-6lr6 krb5kdc[30115](info): commencing operation
krb5kdc: starting...
W20260927 18:43:11.033746 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:43:11 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/rangeradmin_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/rangeradmin_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/rangerlookup_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/rangerlookup_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
I20260927 18:43:11.177268 19374 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJdx3KK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/postgres -l logfile start
2026-09-27 18:43:13.325 UTC [30145] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:43:13.325 UTC [30145] LOG: listening on IPv4 address "127.18.235.148", port 39463
2026-09-27 18:43:13.328 UTC [30145] LOG: listening on Unix socket "/tmp/.s.PGSQL.39463"
2026-09-27 18:43:13.354 UTC [30148] LOG: database system was shut down at 2026-09-27 18:43:12 UTC
2026-09-27 18:43:13.357 UTC [30145] LOG: database system is ready to accept connections
I20260927 18:43:14.221859 19374 mini_postgres.cc:96] Postgres bound to 39463
2026-09-27 18:43:14.227 UTC [30156] FATAL: database "slave" does not exist
127.18.235.148:39463 - accepting connections
I20260927 18:43:14.228010 19374 mini_ranger.cc:161] Starting Ranger...
I20260927 18:43:14.240978 19374 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:43:14.620925 19374 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:43:14.621001 19374 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-admin
2026-09-27 18:43:14,882 [I] DB FLAVOR :POSTGRES
2026-09-27 18:43:14,883 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:43:14,883 [I] Checking connection..
2026-09-27 18:43:14,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:43:15,122 [I] Checking connection passed.
2026-09-27 18:43:15,122 [I] --------- Verifying version history table ---------
2026-09-27 18:43:15,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:43:15,346 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:43:15,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:43:15,558 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:43:15,558 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:43:15,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:43:15.785 UTC [30220] WARNING: there is no transaction in progress
2026-09-27 18:43:15,790 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:43:15,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:43:16,044 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:43:16,045 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:43:16,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:43:16,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:43:16,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6lr6.c.gcp-upstream.internal','N') ;"
2026-09-27 18:43:16,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:43:16,951 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:43:16,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:43:17,170 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:43:17,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:43:17,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:43:17,639 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:43:17,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:43:18.475 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.490 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.504 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.517 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.531 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.546 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.594 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.600 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.613 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.622 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.631 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.643 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.652 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.657 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.662 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:18.668 UTC [30328] WARNING: there is no transaction in progress
2026-09-27 18:43:19,199 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:43:19,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:43:19,435 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:43:19,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:43:19,667 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:43:19,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:43:19,890 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:43:19,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:43:20,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:43:20,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6lr6.c.gcp-upstream.internal';"
2026-09-27 18:43:20,593 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:43:20,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:43:20,835 [I] DB_PATCHES have already been applied
I20260927 18:43:20.840306 19374 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJdx3KK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:43:20.840365 19374 mini_ranger.cc:193] Using host: 127.18.235.148
I20260927 18:43:20.843497 19374 mini_ranger.cc:227] Ranger admin URL: http://127.18.235.148:47869
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:43:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:43:21.820055 19374 mini_ranger.cc:160] Time spent starting Ranger: real 7.592s user 0.006s sys 0.002s
Sep 27, 2026 6:43:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:43:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:43:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:43:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:43:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:43:41.034284 19374 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/rangerkms_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/rangerkms_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.235.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.235.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:43:41 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:43:41.111708 19374 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:43:41.120648 19374 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:43:41.869089 19374 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:43:41.869194 19374 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms
I20260927 18:43:41.869283 19374 mini_ranger_kms.cc:227] Using postgres at 127.18.235.148:39463
2026-09-27 18:43:42,024 [I] DB FLAVOR :POSTGRES
2026-09-27 18:43:42,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:43:42,024 [I] Checking connection
2026-09-27 18:43:42,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:43:42,278 [I] connection success
2026-09-27 18:43:42,278 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:43:42,278 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:43:42,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:43:42,538 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:43:42,538 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:43:42,539 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:43:42,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:39463/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:43:42,803 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:43:43.241150 19374 mini_ranger_kms.cc:310] Created kms service
I20260927 18:43:43.437399 19374 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:43:43.477494 19374 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:43:43 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:43.820461 19374 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:43:43.820604 19374 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:43:43.820627 19374 mini_ranger_kms.cc:240] Using host: 127.18.235.148
I20260927 18:43:43.824410 19374 mini_ranger_kms.cc:276] Ranger KMS PID: 30508
I20260927 18:43:43.824514 19374 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.235.148:54009
Sep 27, 2026 6:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:43:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:43:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:43:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 18:43:45 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534625, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.235.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:45.810527 19374 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.699s user 0.006s sys 0.132s
I20260927 18:43:45.810652 19374 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:43:45.810746 19374 mini_ranger_kms.cc:401] 127.18.235.148:54009/kms/v1/keys
Sep 27 18:43:46 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534625, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.235.148@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
Sep 27 18:43:46 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534625, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.235.148@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
Sep 27 18:43:48 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:43:48.704468 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:33625
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:33625
--ranger_config_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:54009/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:48.809132 30559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:48.809396 30559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:48.809456 30559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:48.813571 30559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:48.813632 30559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:48.813663 30559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:48.813715 30559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:48.813778 30559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:48.820211 30559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.235.148:54009/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:33625
--ranger_config_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:48.821219 30559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:48.822187 30559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:48.830236 30567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:48.830245 30565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:48.830489 30559 server_base.cc:1034] running on GCE node
W20260927 18:43:48.830238 30564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:48.831132 30559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:48.832151 30559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:48.833307 30559 hybrid_clock.cc:648] HybridClock initialized: now 1790534628833290 us; error 32 us; skew 500 ppm
Sep 27 18:43:48 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534628, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:48.835821 30559 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:43:48.836678 30559 webserver.cc:460] Webserver started at http://127.18.235.190:39197/ using document root <none> and password file <none>
I20260927 18:43:48.837190 30559 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:48.837244 30559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:48.837423 30559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:43:48 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534628, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:48.893221 30559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "e6ae5e0d92974c688d79b797689202b9"
format_stamp: "Formatted at 2026-09-27 18:43:48 on dist-test-slave-6lr6"
server_key: "d32265c3a6f0f1bf337cb2d8923b6962"
server_key_iv: "f148b7eee64b1509bc6346188a994c4e"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:48.893797 30559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "e6ae5e0d92974c688d79b797689202b9"
format_stamp: "Formatted at 2026-09-27 18:43:48 on dist-test-slave-6lr6"
server_key: "d32265c3a6f0f1bf337cb2d8923b6962"
server_key_iv: "f148b7eee64b1509bc6346188a994c4e"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:48.897450 30559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:48.912956 30574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:48.914562 30559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:43:48.914688 30559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "e6ae5e0d92974c688d79b797689202b9"
format_stamp: "Formatted at 2026-09-27 18:43:48 on dist-test-slave-6lr6"
server_key: "d32265c3a6f0f1bf337cb2d8923b6962"
server_key_iv: "f148b7eee64b1509bc6346188a994c4e"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:48.914803 30559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:48.950241 30559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:48.953451 30559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:48.953636 30559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:48.962062 30559 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:33625
I20260927 18:43:48.962064 30626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:33625 every 8 connection(s)
I20260927 18:43:48.963213 30559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:48.966022 30627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:48.970654 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 30559
I20260927 18:43:48.970778 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:43:48.971055 19374 external_mini_cluster.cc:1428] Setting key f9084fe98cdadb95195698f2b8114348
I20260927 18:43:48.971655 30627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9: Bootstrap starting.
I20260927 18:43:48.973896 30627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:48.974586 30627 log.cc:824] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:48.976745 30627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9: No bootstrap required, opened a new log
I20260927 18:43:48.980275 30627 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae5e0d92974c688d79b797689202b9" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33625 } }
I20260927 18:43:48.980531 30627 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:48.980609 30627 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ae5e0d92974c688d79b797689202b9, State: Initialized, Role: FOLLOWER
Sep 27 18:43:48 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:43:48.981058 30627 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae5e0d92974c688d79b797689202b9" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33625 } }
I20260927 18:43:48.981228 30627 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:48.981292 30627 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:48.981395 30627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:48.982234 30627 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae5e0d92974c688d79b797689202b9" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33625 } }
I20260927 18:43:48.982568 30627 leader_election.cc:302] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ae5e0d92974c688d79b797689202b9; no voters:
I20260927 18:43:48.982841 30627 leader_election.cc:288] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:48.982960 30632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:48.983176 30632 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [term 1 LEADER]: Becoming Leader. State: Replica: e6ae5e0d92974c688d79b797689202b9, State: Running, Role: LEADER
I20260927 18:43:48.983480 30632 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae5e0d92974c688d79b797689202b9" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33625 } }
I20260927 18:43:48.984030 30627 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:48.984598 30632 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6ae5e0d92974c688d79b797689202b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae5e0d92974c688d79b797689202b9" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33625 } } }
I20260927 18:43:48.984781 30632 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:48.985011 30632 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ae5e0d92974c688d79b797689202b9. Latest consensus state: current_term: 1 leader_uuid: "e6ae5e0d92974c688d79b797689202b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae5e0d92974c688d79b797689202b9" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33625 } } }
I20260927 18:43:48.985162 30632 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:48.985438 30641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:48.987112 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:48.972371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.18.235.190:33625)
0927 18:43:48.972791 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:48.972810 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:48.972852 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:43:48.973708 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:48.973733 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:48.974157 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:48.974529 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:48.975274 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:48.976616 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:48.977867 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:48.978559 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:48.978706 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:48.981558 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:48.981564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:48.981580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:48.981608 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:48.983466 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:48.983960 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:48.983965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:48.983970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:48.984089 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:48.984425 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:48.984428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:48.984429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:48.984888 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:48.985786 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:48.985936 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 18:43:48.986190 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260927 18:43:48.988842 30641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:48.989228 30627 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:43:48.998889 30641 catalog_manager.cc:1347] Generated new cluster ID: 37e7d8b2413d4b8eb04c22e872253dbf
I20260927 18:43:48.999007 30641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:49.008401 30641 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:49.009794 30641 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:49.020622 30641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9: Generated new TSK 0
I20260927 18:43:49.021523 30641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:43:49.702571 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:54009/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33625
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:49.836840 30672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:49.837152 30672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:49.837226 30672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:49.842404 30672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:49.842458 30672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:49.842520 30672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:43:49.848357 30672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.235.148:54009/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:49.849373 30672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:49.850337 30672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:49.861269 30682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:49.861269 30679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:49.861733 30680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:49.861896 30672 server_base.cc:1034] running on GCE node
I20260927 18:43:49.862282 30672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:49.862838 30672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:49.864005 30672 hybrid_clock.cc:648] HybridClock initialized: now 1790534629863971 us; error 41 us; skew 500 ppm
Sep 27 18:43:49 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534629, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:49.866590 30672 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:43:49.867334 30672 webserver.cc:460] Webserver started at http://127.18.235.129:33965/ using document root <none> and password file <none>
I20260927 18:43:49.867823 30672 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:49.867882 30672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:49.868062 30672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:43:49 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534629, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:49.887138 30672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "b394217a5efd469792d3a2db0a09c7f4"
format_stamp: "Formatted at 2026-09-27 18:43:49 on dist-test-slave-6lr6"
server_key: "9511b601f0cc7d385db848fcacf479a2"
server_key_iv: "82b74a473eefadd62981c3f9bb07b5c9"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:49.887704 30672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "b394217a5efd469792d3a2db0a09c7f4"
format_stamp: "Formatted at 2026-09-27 18:43:49 on dist-test-slave-6lr6"
server_key: "9511b601f0cc7d385db848fcacf479a2"
server_key_iv: "82b74a473eefadd62981c3f9bb07b5c9"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:49.891469 30672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:43:49.905416 30689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:49.909727 30672 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260927 18:43:49.909861 30672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "b394217a5efd469792d3a2db0a09c7f4"
format_stamp: "Formatted at 2026-09-27 18:43:49 on dist-test-slave-6lr6"
server_key: "9511b601f0cc7d385db848fcacf479a2"
server_key_iv: "82b74a473eefadd62981c3f9bb07b5c9"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:49.909986 30672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:49.939586 30672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:49.948007 30672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:49.948203 30672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:49.948819 30672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:49.949774 30672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:49.949828 30672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:49.949884 30672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:49.949940 30672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:49.978600 30672 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:39599
I20260927 18:43:49.979771 30672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:49.984503 30804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:39599 every 8 connection(s)
I20260927 18:43:49.987195 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 30672
I20260927 18:43:49.987344 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:49.987663 19374 external_mini_cluster.cc:1428] Setting key bf3b9c2bdae65712779262d686de5388
Sep 27 18:43:49 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534629, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:43:50.005944 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:49.987480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:33625 (local address 127.18.235.129:52689)
0927 18:43:49.989843 (+ 2363us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:49.989900 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:49.990880 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:49.991740 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:49.991749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:49.992096 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:49.992719 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:49.992736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:49.993904 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:49.993908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:49.995308 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:49.995319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:49.996061 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:49.996065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:49.996254 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:49.997999 (+ 1745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:49.998018 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:49.999662 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:50.002779 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:50.002785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:50.002800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:50.003118 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:50.003421 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:50.003425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:50.003427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:50.003597 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:50.004062 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:50.004068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:50.004359 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.004558 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.004875 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"thread_start_us":128,"threads_started":1}
I20260927 18:43:50.006647 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:49.988543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:52689 (local address 127.18.235.190:33625)
0927 18:43:49.991271 (+ 2728us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:49.991280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:49.991292 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:43:49.991328 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:49.991331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:49.991375 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:49.991452 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:49.992888 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:49.993752 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:49.995441 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:49.995800 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:49.995903 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:49.999850 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:49.999853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:49.999856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:49.999880 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:50.002657 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.003227 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.003231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.003232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.003316 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.003701 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.003704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.003706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.003927 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:50.004013 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.006366 (+ 2353us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.006483 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2657,"thread_start_us":86,"threads_started":1}
I20260927 18:43:50.009249 30805 heartbeater.cc:347] Connected to a master server at 127.18.235.190:33625
I20260927 18:43:50.009480 30805 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:50.010030 30805 heartbeater.cc:511] Master 127.18.235.190:33625 requested a full tablet report, sending...
I20260927 18:43:50.011729 30591 ts_manager.cc:194] Registered new tserver with Master: b394217a5efd469792d3a2db0a09c7f4 (127.18.235.129:39599)
I20260927 18:43:50.013286 30591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:52689
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:43:50.045680 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:54009/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33625
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:43:50.206198 30811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:50.206600 30811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:50.206701 30811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:50.212333 30811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:50.212466 30811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:50.212586 30811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:43:50.220211 30811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.235.148:54009/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:50.221653 30811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:50.222942 30811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:50.235450 30818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:50.235908 30820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:50.237025 30811 server_base.cc:1034] running on GCE node
W20260927 18:43:50.241935 30817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:50.242498 30811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:50.243256 30811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:50.244477 30811 hybrid_clock.cc:648] HybridClock initialized: now 1790534630244366 us; error 126 us; skew 500 ppm
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:50.247864 30811 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:43:50.248867 30811 webserver.cc:460] Webserver started at http://127.18.235.130:37167/ using document root <none> and password file <none>
I20260927 18:43:50.249588 30811 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:50.249712 30811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:50.250017 30811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:50.272125 30811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "9f0ccc55304a46488f6409ef2f5612bf"
format_stamp: "Formatted at 2026-09-27 18:43:50 on dist-test-slave-6lr6"
server_key: "21e1f20ccce4a2b7e273746c50608a99"
server_key_iv: "4b07c2fc476e01ddd5786bcec23e0fba"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:50.274407 30811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "9f0ccc55304a46488f6409ef2f5612bf"
format_stamp: "Formatted at 2026-09-27 18:43:50 on dist-test-slave-6lr6"
server_key: "21e1f20ccce4a2b7e273746c50608a99"
server_key_iv: "4b07c2fc476e01ddd5786bcec23e0fba"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:50.279241 30811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 18:43:50.301944 30827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:50.303576 30811 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:50.303694 30811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "9f0ccc55304a46488f6409ef2f5612bf"
format_stamp: "Formatted at 2026-09-27 18:43:50 on dist-test-slave-6lr6"
server_key: "21e1f20ccce4a2b7e273746c50608a99"
server_key_iv: "4b07c2fc476e01ddd5786bcec23e0fba"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:50.303820 30811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:50.322419 30811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:50.325724 30811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:50.325963 30811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:50.326640 30811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:50.327714 30811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:50.327826 30811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:50.327920 30811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:50.327986 30811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:50.341041 30811 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:45715
I20260927 18:43:50.342280 30811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:50.342849 30940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:45715 every 8 connection(s)
I20260927 18:43:50.349936 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 30811
I20260927 18:43:50.350066 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:50.350354 19374 external_mini_cluster.cc:1428] Setting key 0bcbd826e6ce889dc8595e467a4aa0b3
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:43:50.368454 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.346352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:54379 (local address 127.18.235.190:33625)
0927 18:43:50.346466 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.346470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.349560 (+ 3090us) server_negotiation.cc:407] Connection header received
0927 18:43:50.355559 (+ 5999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.355563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.355615 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.355665 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:50.357141 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.357875 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.359552 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.359890 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.359968 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.362873 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:50.362877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:50.362879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:50.362905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:50.364460 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.364901 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.364904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.364906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.364986 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.367238 (+ 2252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.367242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.367244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.367461 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:50.367532 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.368193 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.368302 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:43:50.373860 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.345297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:33625 (local address 127.18.235.130:54379)
0927 18:43:50.349390 (+ 4093us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.349448 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.350494 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.356076 (+ 5582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.356085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.356421 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:50.356973 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.356986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.358016 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.358019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.359410 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.359421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.360171 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.360175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.360392 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.361024 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:50.361044 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:50.362696 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:50.364563 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:50.364569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:50.364583 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:50.364813 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:50.365067 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:50.365070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:50.365073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:50.365265 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:50.367612 (+ 2347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:50.367619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:50.367925 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.368818 (+ 893us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.369148 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3862,"thread_start_us":122,"threads_started":1}
I20260927 18:43:50.375144 30941 heartbeater.cc:347] Connected to a master server at 127.18.235.190:33625
I20260927 18:43:50.375415 30941 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:50.375937 30941 heartbeater.cc:511] Master 127.18.235.190:33625 requested a full tablet report, sending...
I20260927 18:43:50.376957 30591 ts_manager.cc:194] Registered new tserver with Master: 9f0ccc55304a46488f6409ef2f5612bf (127.18.235.130:45715)
I20260927 18:43:50.377540 30591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:54379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:43:50.382333 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:54009/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33625
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:50.502943 30946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:50.503321 30946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:50.503396 30946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:50.507632 30946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:50.507727 30946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:50.507833 30946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:43:50.513020 30946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.235.148:54009/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:33625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:43:50.513888 30946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:50.514770 30946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:50.523655 30954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:50.523919 30946 server_base.cc:1034] running on GCE node
W20260927 18:43:50.524029 30951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:50.523666 30952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:50.524448 30946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:50.524986 30946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:50.526131 30946 hybrid_clock.cc:648] HybridClock initialized: now 1790534630526111 us; error 33 us; skew 500 ppm
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:50.528997 30946 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:43:50.530177 30946 webserver.cc:460] Webserver started at http://127.18.235.131:39917/ using document root <none> and password file <none>
I20260927 18:43:50.530813 30946 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:50.530885 30946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:50.531117 30946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:50.546114 30946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "df468c9eb1e84f539c00760400370c4c"
format_stamp: "Formatted at 2026-09-27 18:43:50 on dist-test-slave-6lr6"
server_key: "b81a67279ca217cb9240c16fccc24adc"
server_key_iv: "bee6f48fcbe4a612bea834d854d374da"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:50.547581 30946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "df468c9eb1e84f539c00760400370c4c"
format_stamp: "Formatted at 2026-09-27 18:43:50 on dist-test-slave-6lr6"
server_key: "b81a67279ca217cb9240c16fccc24adc"
server_key_iv: "bee6f48fcbe4a612bea834d854d374da"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:50.562525 30946 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.000s
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:50.589452 30961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:50.591159 30946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 18:43:50.591284 30946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "df468c9eb1e84f539c00760400370c4c"
format_stamp: "Formatted at 2026-09-27 18:43:50 on dist-test-slave-6lr6"
server_key: "b81a67279ca217cb9240c16fccc24adc"
server_key_iv: "bee6f48fcbe4a612bea834d854d374da"
server_key_version: "kuduclusterkey@0"
I20260927 18:43:50.591408 30946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:50.609612 30946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:50.612938 30946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:50.613178 30946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:50.613780 30946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:50.614733 30946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:50.614820 30946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:50.614898 30946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:50.614945 30946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:50.624595 30946 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:40535
I20260927 18:43:50.624639 31074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:40535 every 8 connection(s)
I20260927 18:43:50.625741 30946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:50.632881 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 30946
I20260927 18:43:50.632989 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:50.633289 19374 external_mini_cluster.cc:1428] Setting key 92304d0db6883de1b86aeb45e6e860f6
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:43:50.640938 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.627586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:33995 (local address 127.18.235.190:33625)
0927 18:43:50.627705 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.627709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.628758 (+ 1049us) server_negotiation.cc:407] Connection header received
0927 18:43:50.629940 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.629944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.629999 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.630087 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:50.631378 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.632212 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.633905 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.634235 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.634332 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.637222 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:50.637225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:50.637228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:50.637256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:50.638836 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.639388 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.639391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.639393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.639469 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.639820 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.639823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.639825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.640030 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:50.640121 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.640680 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.640786 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:43:50.641912 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.628098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:33625 (local address 127.18.235.131:33995)
0927 18:43:50.628609 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.628663 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.629710 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.630269 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.630277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.630591 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:50.631212 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.631229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.632363 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.632366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.633764 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.633775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.634484 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.634488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.634678 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.635389 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:50.635408 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:50.637032 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:50.638958 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:50.638964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:50.638978 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:50.639290 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:50.639552 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:50.639555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:50.639556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:50.639732 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:50.640141 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:50.640147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:50.640431 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.640628 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.640926 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260927 18:43:50.643049 31075 heartbeater.cc:347] Connected to a master server at 127.18.235.190:33625
I20260927 18:43:50.643267 31075 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:50.643795 31075 heartbeater.cc:511] Master 127.18.235.190:33625 requested a full tablet report, sending...
I20260927 18:43:50.644801 30591 ts_manager.cc:194] Registered new tserver with Master: df468c9eb1e84f539c00760400370c4c (127.18.235.131:40535)
I20260927 18:43:50.645539 30591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:33995
I20260927 18:43:50.646756 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:50.666023 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.657461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.18.235.190:33625)
0927 18:43:50.657577 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.657581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.657773 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:43:50.657896 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.657899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.657941 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.658029 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:50.658847 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.659636 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.660793 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.661166 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.661256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.662805 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:50.662809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:50.662812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:50.662837 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:50.664416 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.664864 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.664867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.664869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.664948 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:50.665225 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:50.665228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:50.665230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:50.665420 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:50.665540 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.665743 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.665852 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:43:50.668975 30591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:50.672897 30591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:50.691066 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.682447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:45715 (local address 127.0.0.1:55330)
0927 18:43:50.684560 (+ 2113us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.684605 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.684725 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.685614 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.685621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.685643 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:50.685760 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.685766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.687199 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.687202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.689289 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.689299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.690551 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.690555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.690664 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.690700 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.690796 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.690905 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":89,"threads_started":1}
I20260927 18:43:50.691265 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.683656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:39599 (local address 127.0.0.1:36224)
0927 18:43:50.686154 (+ 2498us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.686165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.686241 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.686960 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.686963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.686974 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:50.687061 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.687066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.688467 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.688471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.689628 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.689640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.690877 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.690881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.690972 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.690995 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.691060 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.691127 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2447,"thread_start_us":73,"threads_started":1}
I20260927 18:43:50.692183 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.684030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.18.235.129:39599)
0927 18:43:50.684321 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.684328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.686338 (+ 2010us) server_negotiation.cc:407] Connection header received
0927 18:43:50.686412 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.686418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.686608 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.686736 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:50.687187 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.688345 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.689773 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.690571 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.690689 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.690812 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.691943 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.692043 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260927 18:43:50.692353 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.682563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.18.235.130:45715)
0927 18:43:50.682870 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.682881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.684846 (+ 1965us) server_negotiation.cc:407] Connection header received
0927 18:43:50.684970 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.684977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.685219 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.685452 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:50.685876 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.687069 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.689441 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.690257 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.690370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.690492 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.691450 (+ 958us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.692217 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260927 18:43:50.695537 30875 tablet_service.cc:1467] Processing CreateTablet for tablet f06d92783f2b4ac4aa94bf681d9d97cf (DEFAULT_TABLE table=test-table [id=d74f782a8d234edf8c82708c18651583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:50.695694 30733 tablet_service.cc:1467] Processing CreateTablet for tablet f06d92783f2b4ac4aa94bf681d9d97cf (DEFAULT_TABLE table=test-table [id=d74f782a8d234edf8c82708c18651583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:50.696748 30733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06d92783f2b4ac4aa94bf681d9d97cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:50.696866 30875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06d92783f2b4ac4aa94bf681d9d97cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:50.697837 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.683467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:40535 (local address 127.0.0.1:59066)
0927 18:43:50.685966 (+ 2499us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.685978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.686873 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.693831 (+ 6958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.693834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.693852 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:50.693930 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.693935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.695373 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.695377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.696444 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.696454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.697497 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.697501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.697585 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.697608 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.697655 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.697715 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2440,"thread_start_us":94,"threads_started":1}
I20260927 18:43:50.698408 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.683854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.18.235.131:40535)
0927 18:43:50.692565 (+ 8711us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.692577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.692601 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:43:50.692674 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.692679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.692876 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.694074 (+ 1198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:50.694209 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.695236 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.696605 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.697365 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.698015 (+ 650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.698131 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.698176 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.698258 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8597,"thread_start_us":80,"threads_started":1}
I20260927 18:43:50.700839 31009 tablet_service.cc:1467] Processing CreateTablet for tablet f06d92783f2b4ac4aa94bf681d9d97cf (DEFAULT_TABLE table=test-table [id=d74f782a8d234edf8c82708c18651583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:50.701786 31009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06d92783f2b4ac4aa94bf681d9d97cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:50.706297 31094 tablet_bootstrap.cc:492] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: Bootstrap starting.
I20260927 18:43:50.707373 31093 tablet_bootstrap.cc:492] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: Bootstrap starting.
I20260927 18:43:50.708570 31094 tablet_bootstrap.cc:654] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:50.709492 31094 log.cc:824] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:50.710064 31093 tablet_bootstrap.cc:654] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:50.710907 31093 log.cc:824] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:50.711441 31094 tablet_bootstrap.cc:492] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: No bootstrap required, opened a new log
I20260927 18:43:50.711621 31094 ts_tablet_manager.cc:1397] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:43:50.712800 31093 tablet_bootstrap.cc:492] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: No bootstrap required, opened a new log
I20260927 18:43:50.712980 31093 ts_tablet_manager.cc:1397] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:43:50.716496 31094 raft_consensus.cc:348] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.716821 31094 raft_consensus.cc:374] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:50.716905 31094 raft_consensus.cc:729] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b394217a5efd469792d3a2db0a09c7f4, State: Initialized, Role: FOLLOWER
I20260927 18:43:50.716761 31093 raft_consensus.cc:348] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.717123 31093 raft_consensus.cc:374] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:50.717216 31093 raft_consensus.cc:729] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0ccc55304a46488f6409ef2f5612bf, State: Initialized, Role: FOLLOWER
I20260927 18:43:50.717443 31094 consensus_queue.cc:260] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.717685 31093 consensus_queue.cc:260] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.718466 31094 ts_tablet_manager.cc:1428] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:43:50.718634 31093 ts_tablet_manager.cc:1428] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260927 18:43:50.718827 30805 heartbeater.cc:503] Master 127.18.235.190:33625 was elected leader, sending a full tablet report...
I20260927 18:43:50.718998 30941 heartbeater.cc:503] Master 127.18.235.190:33625 was elected leader, sending a full tablet report...
I20260927 18:43:50.725741 31099 tablet_bootstrap.cc:492] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Bootstrap starting.
I20260927 18:43:50.728284 31099 tablet_bootstrap.cc:654] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:50.729120 31099 log.cc:824] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:50.731103 31099 tablet_bootstrap.cc:492] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: No bootstrap required, opened a new log
I20260927 18:43:50.731274 31099 ts_tablet_manager.cc:1397] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260927 18:43:50.733860 30806 tablet.cc:2369] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:50.735642 31099 raft_consensus.cc:348] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.735891 31099 raft_consensus.cc:374] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:50.735976 31099 raft_consensus.cc:729] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df468c9eb1e84f539c00760400370c4c, State: Initialized, Role: FOLLOWER
I20260927 18:43:50.736424 31099 consensus_queue.cc:260] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.737071 31075 heartbeater.cc:503] Master 127.18.235.190:33625 was elected leader, sending a full tablet report...
I20260927 18:43:50.737394 31099 ts_tablet_manager.cc:1428] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:43:50.790714 31101 raft_consensus.cc:482] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:50.790942 31101 raft_consensus.cc:504] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.793131 31101 leader_election.cc:288] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f0ccc55304a46488f6409ef2f5612bf (127.18.235.130:45715), b394217a5efd469792d3a2db0a09c7f4 (127.18.235.129:39599)
I20260927 18:43:50.797435 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.792529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:45715 (local address 127.18.235.131:40019)
0927 18:43:50.792776 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.792790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.792868 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.793450 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.793454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.793488 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:50.793592 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.793598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.794678 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.794681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.795913 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.795924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.797008 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.797012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.797141 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.797165 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.797234 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.797305 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260927 18:43:50.797912 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.792968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:48669 (local address 127.18.235.129:39599)
0927 18:43:50.793077 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.793081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.793550 (+ 469us) server_negotiation.cc:407] Connection header received
0927 18:43:50.793719 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.793723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.793765 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.793821 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:50.794222 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.794962 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.796500 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.797169 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.797264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.797312 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.797709 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.797786 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:43:50.798091 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.793044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:40019 (local address 127.18.235.130:45715)
0927 18:43:50.793177 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.793181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.793193 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:43:50.793238 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.793241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.793282 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.793345 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:50.793745 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.794523 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.796086 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.796721 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.796832 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.796885 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.797887 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.797963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:43:50.798652 30889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06d92783f2b4ac4aa94bf681d9d97cf" candidate_uuid: "df468c9eb1e84f539c00760400370c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0ccc55304a46488f6409ef2f5612bf" is_pre_election: true
I20260927 18:43:50.798991 30889 raft_consensus.cc:2454] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df468c9eb1e84f539c00760400370c4c in term 0.
I20260927 18:43:50.799132 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.792582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:39599 (local address 127.18.235.131:48669)
0927 18:43:50.793467 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.793480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:50.793614 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:50.793955 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:50.793958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:50.793974 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:50.794085 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.794092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.795105 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.795108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.796367 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:50.796378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.797461 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.797465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:50.797560 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.797583 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:43:50.798337 (+ 754us) client_negotiation.cc:240] Negotiation successful
0927 18:43:50.798981 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":109,"threads_started":1}
I20260927 18:43:50.799525 30965 leader_election.cc:302] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0ccc55304a46488f6409ef2f5612bf, df468c9eb1e84f539c00760400370c4c; no voters:
I20260927 18:43:50.799853 31101 raft_consensus.cc:2781] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:50.799922 31101 raft_consensus.cc:482] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:50.799957 31101 raft_consensus.cc:3037] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:50.799984 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06d92783f2b4ac4aa94bf681d9d97cf" candidate_uuid: "df468c9eb1e84f539c00760400370c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b394217a5efd469792d3a2db0a09c7f4" is_pre_election: true
I20260927 18:43:50.800237 30759 raft_consensus.cc:2454] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df468c9eb1e84f539c00760400370c4c in term 0.
I20260927 18:43:50.801209 31101 raft_consensus.cc:504] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.801570 31101 leader_election.cc:288] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [CANDIDATE]: Term 1 election: Requested vote from peers 9f0ccc55304a46488f6409ef2f5612bf (127.18.235.130:45715), b394217a5efd469792d3a2db0a09c7f4 (127.18.235.129:39599)
I20260927 18:43:50.801937 30889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06d92783f2b4ac4aa94bf681d9d97cf" candidate_uuid: "df468c9eb1e84f539c00760400370c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0ccc55304a46488f6409ef2f5612bf"
I20260927 18:43:50.802014 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06d92783f2b4ac4aa94bf681d9d97cf" candidate_uuid: "df468c9eb1e84f539c00760400370c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b394217a5efd469792d3a2db0a09c7f4"
I20260927 18:43:50.802052 30889 raft_consensus.cc:3037] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:50.802119 30759 raft_consensus.cc:3037] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:50.803277 30889 raft_consensus.cc:2454] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df468c9eb1e84f539c00760400370c4c in term 1.
I20260927 18:43:50.803372 30759 raft_consensus.cc:2454] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df468c9eb1e84f539c00760400370c4c in term 1.
I20260927 18:43:50.803680 30962 leader_election.cc:302] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b394217a5efd469792d3a2db0a09c7f4, df468c9eb1e84f539c00760400370c4c; no voters:
I20260927 18:43:50.803859 31101 raft_consensus.cc:2781] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:50.804091 31101 raft_consensus.cc:686] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 1 LEADER]: Becoming Leader. State: Replica: df468c9eb1e84f539c00760400370c4c, State: Running, Role: LEADER
I20260927 18:43:50.804672 31101 consensus_queue.cc:237] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } }
I20260927 18:43:50.808260 30590 catalog_manager.cc:5495] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c reported cstate change: term changed from 0 to 1, leader changed from <none> to df468c9eb1e84f539c00760400370c4c (127.18.235.131). New cstate: current_term: 1 leader_uuid: "df468c9eb1e84f539c00760400370c4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df468c9eb1e84f539c00760400370c4c" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 } health_report { overall_health: UNKNOWN } } }
I20260927 18:43:50.843796 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.839536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.18.235.131:40535)
0927 18:43:50.839656 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:50.839660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:50.839676 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:43:50.839866 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:50.839870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:50.839925 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:50.840013 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:50.840254 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.840752 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.841675 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:50.842004 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:50.842105 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:50.842541 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:50.843090 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:50.843204 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:50.843593 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:43:50.843654 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:43:50.847164 30942 tablet.cc:2369] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:50.852576 30759 raft_consensus.cc:1261] T f06d92783f2b4ac4aa94bf681d9d97cf P b394217a5efd469792d3a2db0a09c7f4 [term 1 FOLLOWER]: Refusing update from remote peer df468c9eb1e84f539c00760400370c4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:50.852995 30889 raft_consensus.cc:1261] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 1 FOLLOWER]: Refusing update from remote peer df468c9eb1e84f539c00760400370c4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:50.853786 31101 consensus_queue.cc:1035] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b394217a5efd469792d3a2db0a09c7f4" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:50.854228 31103 consensus_queue.cc:1035] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0ccc55304a46488f6409ef2f5612bf" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 45715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 18:43:50 dist-test-slave-6lr6 krb5kdc[30115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:43:50.870708 31107 mvcc.cc:205] Tried to move back new op lower bound from 7334029847964708864 to 7334029847779954688. Current Snapshot: MvccSnapshot[applied={T|T < 7334029847964708864}]
I20260927 18:43:50.871078 31110 mvcc.cc:205] Tried to move back new op lower bound from 7334029847964708864 to 7334029847779954688. Current Snapshot: MvccSnapshot[applied={T|T < 7334029847964708864}]
I20260927 18:43:50.880609 31108 mvcc.cc:205] Tried to move back new op lower bound from 7334029847964708864 to 7334029847779954688. Current Snapshot: MvccSnapshot[applied={T|T < 7334029847964708864}]
I20260927 18:43:50.883474 30591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:50.883648 30591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:50.886467 30591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6ae5e0d92974c688d79b797689202b9: Sending DeleteTablet for 3 replicas of tablet f06d92783f2b4ac4aa94bf681d9d97cf
I20260927 18:43:50.887512 30875 tablet_service.cc:1514] Processing DeleteTablet for tablet f06d92783f2b4ac4aa94bf681d9d97cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:50 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:55330
I20260927 18:43:50.887982 30733 tablet_service.cc:1514] Processing DeleteTablet for tablet f06d92783f2b4ac4aa94bf681d9d97cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:50 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:36224
I20260927 18:43:50.888868 31009 tablet_service.cc:1514] Processing DeleteTablet for tablet f06d92783f2b4ac4aa94bf681d9d97cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:50 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:59066
I20260927 18:43:50.890049 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 30672
I20260927 18:43:50.896607 31116 tablet_replica.cc:330] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf: stopping tablet replica
I20260927 18:43:50.896517 31118 tablet_replica.cc:330] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: stopping tablet replica
I20260927 18:43:50.897040 31118 raft_consensus.cc:2229] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:50.897470 31116 raft_consensus.cc:2229] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:50.897836 31116 raft_consensus.cc:2258] T f06d92783f2b4ac4aa94bf681d9d97cf P 9f0ccc55304a46488f6409ef2f5612bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:50.898188 31118 raft_consensus.cc:2258] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:50.899917 31118 ts_tablet_manager.cc:1905] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:50.903885 31118 ts_tablet_manager.cc:1918] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 18:43:50.906000 30575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:39599 (error 108)
I20260927 18:43:50.906767 31118 log.cc:1196] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/f06d92783f2b4ac4aa94bf681d9d97cf
I20260927 18:43:50.907006 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 30811
I20260927 18:43:50.910686 31118 ts_tablet_manager.cc:1939] T f06d92783f2b4ac4aa94bf681d9d97cf P df468c9eb1e84f539c00760400370c4c: Deleting consensus metadata
I20260927 18:43:50.911890 30576 catalog_manager.cc:4841] TS df468c9eb1e84f539c00760400370c4c (127.18.235.131:40535): tablet f06d92783f2b4ac4aa94bf681d9d97cf (table test-table [id=d74f782a8d234edf8c82708c18651583]) successfully deleted
I20260927 18:43:50.921788 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.921488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:39599 (local address 127.0.0.1:36230)
0927 18:43:50.921620 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.921700 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:39599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:43:50.922092 30575 catalog_manager.cc:4584] TS b394217a5efd469792d3a2db0a09c7f4 (127.18.235.129:39599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f06d92783f2b4ac4aa94bf681d9d97cf: Network error: Client connection negotiation failed: client connection to 127.18.235.129:39599: connect: Connection refused (error 111)
I20260927 18:43:50.927659 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:50.927408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:45715 (local address 127.0.0.1:55340)
0927 18:43:50.927519 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:50.927577 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.130:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:43:50.927675 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 30946
I20260927 18:43:50.934454 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 30559
I20260927 18:43:50.940543 19374 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:43:51.284595 19374 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:43:51.284720 19374 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:43:51.618170 19374 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:43:51.618 UTC [30145] LOG: received smart shutdown request
2026-09-27 18:43:51.621 UTC [30145] LOG: background worker "logical replication launcher" (PID 30154) exited with exit code 1
2026-09-27 18:43:51.621 UTC [30149] LOG: shutting down
2026-09-27 18:43:51.775 UTC [30145] LOG: database system is shut down
2026-09-27T18:43:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41786 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:51 dist-test-slave-6lr6 krb5kdc[31129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:51 dist-test-slave-6lr6 krb5kdc[31129](info): set up 2 sockets
Sep 27 18:43:51 dist-test-slave-6lr6 krb5kdc[31129](info): commencing operation
krb5kdc: starting...
W20260927 18:43:52.812909 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:43:52 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/rangeradmin_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/rangeradmin_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/rangerlookup_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/rangerlookup_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
I20260927 18:43:52.960767 19374 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJdx3KK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/postgres -l logfile start
2026-09-27 18:43:55.107 UTC [31159] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:43:55.107 UTC [31159] LOG: listening on IPv4 address "127.18.235.148", port 49441
2026-09-27 18:43:55.110 UTC [31159] LOG: listening on Unix socket "/tmp/.s.PGSQL.49441"
2026-09-27 18:43:55.135 UTC [31162] LOG: database system was shut down at 2026-09-27 18:43:53 UTC
2026-09-27 18:43:55.138 UTC [31159] LOG: database system is ready to accept connections
I20260927 18:43:56.004587 19374 mini_postgres.cc:96] Postgres bound to 49441
2026-09-27 18:43:56.010 UTC [31170] FATAL: database "slave" does not exist
127.18.235.148:49441 - accepting connections
I20260927 18:43:56.010849 19374 mini_ranger.cc:161] Starting Ranger...
I20260927 18:43:56.024154 19374 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:43:56.412890 19374 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:43:56.412954 19374 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-admin
2026-09-27 18:43:56,692 [I] DB FLAVOR :POSTGRES
2026-09-27 18:43:56,693 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:43:56,693 [I] Checking connection..
2026-09-27 18:43:56,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:43:56,911 [I] Checking connection passed.
2026-09-27 18:43:56,911 [I] --------- Verifying version history table ---------
2026-09-27 18:43:56,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:43:57,133 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:43:57,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:43:57,368 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:43:57,368 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:43:57,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:43:57.582 UTC [31234] WARNING: there is no transaction in progress
2026-09-27 18:43:57,587 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:43:57,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:43:57,835 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:43:57,835 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:43:57,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:43:58,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:43:58,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6lr6.c.gcp-upstream.internal','N') ;"
2026-09-27 18:43:58,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:43:58,775 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:43:58,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:43:59,005 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:43:59,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:43:59,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:43:59,460 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:43:59,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:44:00.320 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.339 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.353 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.366 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.379 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.396 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.455 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.461 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.473 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.482 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.491 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.503 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.511 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.516 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.521 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:00.526 UTC [31342] WARNING: there is no transaction in progress
2026-09-27 18:44:01,037 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:44:01,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:44:01,242 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:44:01,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:44:01,459 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:44:01,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:44:01,673 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:44:01,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:44:01,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:44:02,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6lr6.c.gcp-upstream.internal';"
2026-09-27 18:44:02,379 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:44:02,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:44:02,615 [I] DB_PATCHES have already been applied
I20260927 18:44:02.620970 19374 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJdx3KK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:44:02.621023 19374 mini_ranger.cc:193] Using host: 127.18.235.148
I20260927 18:44:02.624055 19374 mini_ranger.cc:227] Ranger admin URL: http://127.18.235.148:52493
Sep 27, 2026 6:44:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:44:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:44:03.603216 19374 mini_ranger.cc:160] Time spent starting Ranger: real 7.592s user 0.007s sys 0.001s
Sep 27, 2026 6:44:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:44:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 6:44:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:44:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:44:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:44:24.129035 19374 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/rangerkms_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/rangerkms_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.235.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.235.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.235.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/HTTP_127.18.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:44:24 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534664, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:44:24.201004 19374 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:44:24.209388 19374 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:44:24.939381 19374 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:44:24.939471 19374 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms
I20260927 18:44:24.939491 19374 mini_ranger_kms.cc:227] Using postgres at 127.18.235.148:49441
2026-09-27 18:44:25,086 [I] DB FLAVOR :POSTGRES
2026-09-27 18:44:25,087 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:44:25,087 [I] Checking connection
2026-09-27 18:44:25,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:44:25,306 [I] connection success
2026-09-27 18:44:25,306 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:44:25,307 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:44:25,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:44:25,521 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:44:25,522 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:44:25,522 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:44:25,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.235.148:49441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:44:25,776 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:44:26.190476 19374 mini_ranger_kms.cc:310] Created kms service
I20260927 18:44:26.372706 19374 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:44:26.410112 19374 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:44:26 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534664, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:26.774147 19374 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:44:26.774310 19374 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJdx3KK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJdx3KK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:44:26.774333 19374 mini_ranger_kms.cc:240] Using host: 127.18.235.148
I20260927 18:44:26.777859 19374 mini_ranger_kms.cc:276] Ranger KMS PID: 31522
I20260927 18:44:26.777943 19374 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.235.148:42583
Sep 27, 2026 6:44:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:44:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:44:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:44:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:44:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:44:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 18:44:28 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.235.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:28.805703 19374 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.605s user 0.028s sys 0.104s
I20260927 18:44:28.805824 19374 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:44:28.805888 19374 mini_ranger_kms.cc:401] 127.18.235.148:42583/kms/v1/keys
Sep 27 18:44:29 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.235.148@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
Sep 27 18:44:29 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.235.148@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
Sep 27 18:44:31 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:44:31.577878 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45999
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:45999
--ranger_config_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:31.679807 31573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:31.680027 31573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:31.680063 31573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:31.684003 31573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:31.684051 31573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:31.684068 31573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:31.684082 31573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:31.684096 31573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:31.689244 31573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.235.148:42583/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:45999
--ranger_config_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:31.690119 31573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:31.691030 31573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:31.698686 31578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:31.698678 31579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:31.698946 31581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:31.698802 31573 server_base.cc:1034] running on GCE node
I20260927 18:44:31.699492 31573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:31.700461 31573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:31.701617 31573 hybrid_clock.cc:648] HybridClock initialized: now 1790534671701600 us; error 33 us; skew 500 ppm
Sep 27 18:44:31 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:31.704262 31573 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:31.705296 31573 webserver.cc:460] Webserver started at http://127.18.235.190:37781/ using document root <none> and password file <none>
I20260927 18:44:31.705951 31573 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:31.706029 31573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:31.706259 31573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:44:31 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:31.758147 31573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "4f87978cb6c44ea59c1fd86d622d96ce"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-6lr6"
server_key: "f73777c7d6d4fe492ccad79cae7cc5d6"
server_key_iv: "18acffda97b119337b92f333509317c2"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:31.758867 31573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "4f87978cb6c44ea59c1fd86d622d96ce"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-6lr6"
server_key: "f73777c7d6d4fe492ccad79cae7cc5d6"
server_key_iv: "18acffda97b119337b92f333509317c2"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:31.763494 31573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:44:31.776746 31588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.778067 31573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:31.778187 31573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "4f87978cb6c44ea59c1fd86d622d96ce"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-6lr6"
server_key: "f73777c7d6d4fe492ccad79cae7cc5d6"
server_key_iv: "18acffda97b119337b92f333509317c2"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:31.778299 31573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:31.791071 31573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:31.793942 31573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:31.794112 31573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:31.801975 31640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:45999 every 8 connection(s)
I20260927 18:44:31.801982 31573 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:45999
I20260927 18:44:31.803125 31573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:31.805883 31641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:31.810988 31641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Bootstrap starting.
I20260927 18:44:31.813282 31641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:31.813313 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 31573
I20260927 18:44:31.813421 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:44:31.813694 19374 external_mini_cluster.cc:1428] Setting key dd1d5dedfcfed46306e0fdb68456effc
I20260927 18:44:31.814121 31641 log.cc:824] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:31.815944 31641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: No bootstrap required, opened a new log
I20260927 18:44:31.818958 31641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:31.819154 31641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:31.819229 31641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f87978cb6c44ea59c1fd86d622d96ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:31.819645 31641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:31.819768 31641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:31.819842 31641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:31.819973 31641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:31.820922 31641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:31.821250 31641 leader_election.cc:302] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f87978cb6c44ea59c1fd86d622d96ce; no voters:
I20260927 18:44:31.821523 31641 leader_election.cc:288] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:31.821586 31646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:31.821847 31646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 LEADER]: Becoming Leader. State: Replica: 4f87978cb6c44ea59c1fd86d622d96ce, State: Running, Role: LEADER
I20260927 18:44:31.822160 31646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:31.822768 31641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 18:44:31 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:31.824837 31641 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:44:31.823706 31647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } } }
I20260927 18:44:31.825090 31647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:31.823318 31646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f87978cb6c44ea59c1fd86d622d96ce. Latest consensus state: current_term: 1 leader_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } } }
I20260927 18:44:31.825865 31646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:31.828685 31660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:31.832481 31660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:31.832623 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:31.814760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.18.235.190:45999)
0927 18:44:31.815222 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:31.815241 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:31.815283 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:44:31.816103 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:31.816126 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:31.816535 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:31.816875 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:31.817720 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.819729 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.820919 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.821642 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.823458 (+ 1816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:31.824645 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:31.824652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:31.824667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:31.824696 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:31.828666 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:31.829285 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:31.829291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:31.829297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:31.829493 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:31.829807 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:31.829810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:31.829812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:31.830589 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:31.831283 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:31.831442 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 18:44:31.831746 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":112,"threads_started":1}
I20260927 18:44:31.843091 31660 catalog_manager.cc:1347] Generated new cluster ID: b96c22bf170a4ddc88c66f68b0adfc36
I20260927 18:44:31.843148 31660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:31.874301 31660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:31.875633 31660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:31.884644 31660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Generated new TSK 0
I20260927 18:44:31.885504 31660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:44:32.523275 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45999
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:32.675103 31686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:32.675318 31686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:32.675355 31686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:32.679399 31686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:32.679446 31686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:32.679500 31686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:44:32.684661 31686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.235.148:42583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:32.685587 31686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:32.686749 31686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:32.696355 31694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:32.696645 31693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:32.696791 31686 server_base.cc:1034] running on GCE node
W20260927 18:44:32.696848 31696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:32.697284 31686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:32.697912 31686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:32.705240 31686 hybrid_clock.cc:648] HybridClock initialized: now 1790534672705213 us; error 37 us; skew 500 ppm
Sep 27 18:44:32 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:32.707705 31686 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:44:32.708473 31686 webserver.cc:460] Webserver started at http://127.18.235.129:36639/ using document root <none> and password file <none>
I20260927 18:44:32.709019 31686 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:32.709084 31686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:32.709282 31686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:44:32 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:32.723569 31686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "c8100190a1204be780ac386ca6746f3e"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-6lr6"
server_key: "af9f25aaf1eb65ba168a3839b8e4ee35"
server_key_iv: "bea34665bfea1fd81e92a19f318b044a"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:32.724292 31686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "c8100190a1204be780ac386ca6746f3e"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-6lr6"
server_key: "af9f25aaf1eb65ba168a3839b8e4ee35"
server_key_iv: "bea34665bfea1fd81e92a19f318b044a"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:32.728703 31686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:32.745296 31703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:32.747802 31686 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260927 18:44:32.747923 31686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "c8100190a1204be780ac386ca6746f3e"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-6lr6"
server_key: "af9f25aaf1eb65ba168a3839b8e4ee35"
server_key_iv: "bea34665bfea1fd81e92a19f318b044a"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:32.748041 31686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:32.769690 31686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:32.772790 31686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:32.772993 31686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:32.773666 31686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:32.774649 31686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:32.774744 31686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:32.774824 31686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:32.774871 31686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:32.784622 31686 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:33545
I20260927 18:44:32.785660 31686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:32.789549 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 31686
I20260927 18:44:32.789654 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:32.789956 19374 external_mini_cluster.cc:1428] Setting key 85b50f80dbc14f903ca0121392cec41f
I20260927 18:44:32.792584 31816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:33545 every 8 connection(s)
Sep 27 18:44:32 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:44:32.820178 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.794649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45999 (local address 127.18.235.129:54287)
0927 18:44:32.795123 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:32.795172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:32.796095 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:32.797765 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:32.797774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:32.798107 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:32.798618 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:32.798631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.799585 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.799588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:32.800454 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:32.800463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.800930 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.800943 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:32.801088 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.801747 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:32.801846 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:32.806148 (+ 4302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:32.816497 (+ 10349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:32.816503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:32.816517 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:32.816827 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:32.817536 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:32.817539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:32.817541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:32.817711 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:32.818303 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:32.818309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:32.818610 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 18:44:32.818795 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 18:44:32.819166 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:44:32.821602 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.795649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:54287 (local address 127.18.235.190:45999)
0927 18:44:32.797287 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.797296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.797310 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:44:32.797349 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.797352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.797397 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.797487 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.798756 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.799443 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.800573 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.800801 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.800881 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.813192 (+ 12311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.813195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.813198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.813226 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.816347 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.816950 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.816953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.816955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.817046 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.817822 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.817825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.817827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.818042 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.821246 (+ 3204us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.821322 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.821449 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":99,"threads_started":1}
I20260927 18:44:32.824460 31817 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45999
I20260927 18:44:32.824698 31817 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:32.825264 31817 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:32.826937 31605 ts_manager.cc:194] Registered new tserver with Master: c8100190a1204be780ac386ca6746f3e (127.18.235.129:33545)
I20260927 18:44:32.828347 31605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:54287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:44:32.839752 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45999
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:44:32 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:44:32.992606 31825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:32.992983 31825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:32.993062 31825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:32.999059 31825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:32.999141 31825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:32.999246 31825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:44:33.006891 31825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.235.148:42583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:33.008344 31825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:33.009701 31825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:33.022650 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.022739 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.023617 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:33.025046 31825 server_base.cc:1034] running on GCE node
I20260927 18:44:33.025508 31825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:33.026242 31825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:33.027427 31825 hybrid_clock.cc:648] HybridClock initialized: now 1790534673027399 us; error 43 us; skew 500 ppm
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:33.030908 31825 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:44:33.031971 31825 webserver.cc:460] Webserver started at http://127.18.235.130:38849/ using document root <none> and password file <none>
I20260927 18:44:33.032655 31825 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:33.032742 31825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:33.032971 31825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:33.048275 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "f0c4f6e05a090178e1976239f4a019c6"
server_key_iv: "b3c633863b45471a0d6fc42b53fb818d"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.048986 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "f0c4f6e05a090178e1976239f4a019c6"
server_key_iv: "b3c633863b45471a0d6fc42b53fb818d"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.054062 31825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260927 18:44:33.089160 31841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.090883 31825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:33.091022 31825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "f0c4f6e05a090178e1976239f4a019c6"
server_key_iv: "b3c633863b45471a0d6fc42b53fb818d"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.091167 31825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:33.124784 31825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:33.130316 31825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:33.130539 31825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:33.131300 31825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:33.132375 31825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:33.132452 31825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.132524 31825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:33.132560 31825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.150271 31825 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:38095
I20260927 18:44:33.151386 31825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:33.152287 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 31825
I20260927 18:44:33.152410 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:33.152666 19374 external_mini_cluster.cc:1428] Setting key daeedcca70232b52cbbd4813de8a33ec
I20260927 18:44:33.161885 31954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:38095 every 8 connection(s)
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:33.193840 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:33.167937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:59461 (local address 127.18.235.190:45999)
0927 18:44:33.168063 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:33.168068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:33.169795 (+ 1727us) server_negotiation.cc:407] Connection header received
0927 18:44:33.173503 (+ 3708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:33.173508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:33.173572 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:33.173634 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:33.175400 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.176259 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.178130 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.178522 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.178608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:33.184361 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:33.184365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:33.184369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:33.184399 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:33.186284 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:33.186969 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:33.186973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:33.186975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:33.187074 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:33.187556 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:33.187560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:33.187562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:33.187805 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:33.187901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:33.193418 (+ 5517us) server_negotiation.cc:299] Negotiation successful
0927 18:44:33.193583 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:44:33.195983 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:33.166793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45999 (local address 127.18.235.130:59461)
0927 18:44:33.169616 (+ 2823us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:33.169676 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:33.171113 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:33.174084 (+ 2971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:33.174095 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:33.174488 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:33.175193 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:33.175216 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.176451 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.176454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:33.177988 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:33.178000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.178825 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.178829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:33.179023 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:33.182209 (+ 3186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:33.182234 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:33.184133 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:33.186462 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:33.186468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:33.186485 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:33.186839 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:33.187194 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:33.187197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:33.187200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:33.187408 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:33.187974 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:33.187981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:33.188298 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 18:44:33.194389 (+ 6091us) client_negotiation.cc:240] Negotiation successful
0927 18:44:33.194800 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2578,"thread_start_us":158,"threads_started":1}
I20260927 18:44:33.197852 31955 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45999
I20260927 18:44:33.198138 31955 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:33.198742 31955 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:33.200021 31605 ts_manager.cc:194] Registered new tserver with Master: 63e94e23c7ad42d38d5b7b2a4fa9ae67 (127.18.235.130:38095)
I20260927 18:44:33.200755 31605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:59461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:44:33.210045 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45999
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
W20260927 18:44:33.364637 31960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:33.364965 31960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:33.365033 31960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:33.371037 31960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:33.371104 31960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:33.371176 31960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:44:33.378059 31960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.235.148:42583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:33.379357 31960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:33.380563 31960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:33.392215 31966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.392529 31968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:33.393595 31960 server_base.cc:1034] running on GCE node
W20260927 18:44:33.393879 31965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:33.394367 31960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:33.395068 31960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:33.396234 31960 hybrid_clock.cc:648] HybridClock initialized: now 1790534673396174 us; error 72 us; skew 500 ppm
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:33.399636 31960 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:44:33.400647 31960 webserver.cc:460] Webserver started at http://127.18.235.131:41167/ using document root <none> and password file <none>
I20260927 18:44:33.401351 31960 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:33.401437 31960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:33.401659 31960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:33.424543 31960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "ef4cf75a54334915bdba42e78573b060"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "a37e09ee3f59ea89f18b836eecf9881e"
server_key_iv: "3bc7f06b359549edc1b0638e7f462dfa"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.425292 31960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "ef4cf75a54334915bdba42e78573b060"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "a37e09ee3f59ea89f18b836eecf9881e"
server_key_iv: "3bc7f06b359549edc1b0638e7f462dfa"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.429787 31960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:33.448827 31975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.454388 31960 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260927 18:44:33.454512 31960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "ef4cf75a54334915bdba42e78573b060"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "a37e09ee3f59ea89f18b836eecf9881e"
server_key_iv: "3bc7f06b359549edc1b0638e7f462dfa"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.454633 31960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:33.480952 31960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:33.484316 31960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:33.484488 31960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:33.485157 31960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:33.486101 31960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:33.486158 31960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.486207 31960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:33.486234 31960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.495658 31960 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:37673
I20260927 18:44:33.496788 31960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:33.500771 32088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:37673 every 8 connection(s)
I20260927 18:44:33.502424 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 31960
I20260927 18:44:33.502552 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:33.502797 19374 external_mini_cluster.cc:1428] Setting key 895423c41573c0a3dba1a944c6d3a234
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:33.519112 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:33.500439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:38783 (local address 127.18.235.190:45999)
0927 18:44:33.500577 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:33.500582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:33.501524 (+ 942us) server_negotiation.cc:407] Connection header received
0927 18:44:33.503022 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:33.503026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:33.503086 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:33.503146 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:33.506804 (+ 3658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.507692 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.509546 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.509931 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.510013 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:33.513055 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:33.513059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:33.513062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:33.513091 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:33.514916 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:33.517202 (+ 2286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:33.517206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:33.517208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:33.517312 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:33.517763 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:33.517766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:33.517768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:33.518020 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:33.518112 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:33.518790 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 18:44:33.518929 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:44:33.520917 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:33.499313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45999 (local address 127.18.235.131:38783)
0927 18:44:33.501354 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:33.501415 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:33.502678 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:33.505564 (+ 2886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:33.505574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:33.505942 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:33.506626 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:33.506647 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.507970 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.507974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:33.509393 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:33.509405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.510238 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.510242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:33.510441 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:33.511093 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:33.511116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:33.512831 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:33.515066 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:33.515072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:33.515087 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:33.517052 (+ 1965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:33.517442 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:33.517445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:33.517447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:33.517632 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:33.518185 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:33.518192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:33.518493 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 18:44:33.519554 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0927 18:44:33.519900 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1474,"thread_start_us":143,"threads_started":1}
I20260927 18:44:33.522374 32089 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45999
I20260927 18:44:33.522603 32089 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:33.523185 32089 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:33.524423 31605 ts_manager.cc:194] Registered new tserver with Master: ef4cf75a54334915bdba42e78573b060 (127.18.235.131:37673)
I20260927 18:44:33.525244 31605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:38783
I20260927 18:44:33.529176 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:44:33.529966 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 31686
I20260927 18:44:33.537374 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 31825
I20260927 18:44:33.543885 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 31960
I20260927 18:44:33.550029 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 31573
I20260927 18:44:33.555107 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45999
--webserver_interface=127.18.235.190
--webserver_port=37781
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:45999
--ranger_config_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:33.657787 32092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:33.658003 32092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:33.658038 32092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:33.661921 32092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:33.661965 32092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:33.661983 32092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:33.661996 32092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:33.662009 32092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:33.666998 32092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:45999
--ranger_config_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=37781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:33.667837 32092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:33.668704 32092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:33.676409 32097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.676421 32098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.676403 32100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:33.676800 32092 server_base.cc:1034] running on GCE node
I20260927 18:44:33.677318 32092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:33.678246 32092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:33.679409 32092 hybrid_clock.cc:648] HybridClock initialized: now 1790534673679389 us; error 35 us; skew 500 ppm
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:33.681763 32092 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:33.682574 32092 webserver.cc:460] Webserver started at http://127.18.235.190:37781/ using document root <none> and password file <none>
I20260927 18:44:33.683027 32092 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:33.683084 32092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:33.740095 32092 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:44:33.742309 32108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.743479 32092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:44:33.743590 32092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "4f87978cb6c44ea59c1fd86d622d96ce"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-6lr6"
server_key: "f73777c7d6d4fe492ccad79cae7cc5d6"
server_key_iv: "18acffda97b119337b92f333509317c2"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.744148 32092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:33.763711 32092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:33.766253 32092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:33.766439 32092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:33.773363 32092 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:45999
I20260927 18:44:33.773373 32160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:45999 every 8 connection(s)
I20260927 18:44:33.774364 32092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:33.778475 32161 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:44:33.779997 32161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Bootstrap starting.
I20260927 18:44:33.781011 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 32092
I20260927 18:44:33.781458 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:33545
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:45999
--webserver_port=36639
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:33.795182 32161 log.cc:824] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:33.801035 32161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:44:33.801370 32161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Bootstrap complete.
I20260927 18:44:33.804943 32161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:33.805331 32161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f87978cb6c44ea59c1fd86d622d96ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:33.805823 32161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:33.805939 32161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:33.806003 32161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:33.806090 32161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:44:33.807742 32161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:33.808027 32161 leader_election.cc:302] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f87978cb6c44ea59c1fd86d622d96ce; no voters:
I20260927 18:44:33.808282 32161 leader_election.cc:288] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:44:33.808382 32165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:44:33.808646 32165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [term 2 LEADER]: Becoming Leader. State: Replica: 4f87978cb6c44ea59c1fd86d622d96ce, State: Running, Role: LEADER
I20260927 18:44:33.808982 32165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } }
I20260927 18:44:33.809396 32161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:33.810376 32166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } } }
I20260927 18:44:33.810504 32166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:33.810901 32167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f87978cb6c44ea59c1fd86d622d96ce. Latest consensus state: current_term: 2 leader_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f87978cb6c44ea59c1fd86d622d96ce" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45999 } } }
I20260927 18:44:33.811045 32167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:33.811337 32175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:33.811472 32161 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:44:33.817329 32175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:33.817989 32175 catalog_manager.cc:1259] Loaded cluster ID: b96c22bf170a4ddc88c66f68b0adfc36
I20260927 18:44:33.818053 32175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:33.820540 32175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:33.821242 32175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Loaded TSK: 0
I20260927 18:44:33.822422 32175 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:44:33.891719 32163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:33.891994 32163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:33.892074 32163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:33.896411 32163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:33.896477 32163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:33.896559 32163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:44:33.901779 32163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:33545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=36639
--tserver_master_addrs=127.18.235.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:33.902635 32163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:33.903509 32163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:33.914247 32201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.914772 32198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:33.914942 32199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:33.915388 32163 server_base.cc:1034] running on GCE node
I20260927 18:44:33.915740 32163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:33.916334 32163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:33.917486 32163 hybrid_clock.cc:648] HybridClock initialized: now 1790534673917464 us; error 40 us; skew 500 ppm
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:33.920032 32163 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:44:33.920825 32163 webserver.cc:460] Webserver started at http://127.18.235.129:36639/ using document root <none> and password file <none>
I20260927 18:44:33.921350 32163 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:33.921427 32163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:44:33 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:33.947638 32163 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.002s sys 0.000s
I20260927 18:44:33.950333 32209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.951695 32163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:44:33.951815 32163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "c8100190a1204be780ac386ca6746f3e"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-6lr6"
server_key: "af9f25aaf1eb65ba168a3839b8e4ee35"
server_key_iv: "bea34665bfea1fd81e92a19f318b044a"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:33.952323 32163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:33.974922 32163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:33.977655 32163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:33.977818 32163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:33.978358 32163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:33.979180 32163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:33.979230 32163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.979280 32163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:33.979303 32163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:33.989038 32163 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:33545
I20260927 18:44:33.989085 32322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:33545 every 8 connection(s)
I20260927 18:44:33.990533 32163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:34.003288 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 32163
I20260927 18:44:34.003760 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:38095
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:45999
--webserver_port=38849
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534673, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:34.011344 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:33.992665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45999 (local address 127.18.235.129:37233)
0927 18:44:33.993965 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:33.994004 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:33.994820 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:33.996602 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:33.996611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:33.996934 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:33.997559 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:33.997578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.999886 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:33.999890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:34.001221 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:34.001233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.002406 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.002410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:34.002587 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.003124 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:34.003144 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:34.005971 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:34.008233 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:34.008238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:34.008252 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:34.008565 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:34.008979 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:34.008982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:34.008984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:34.009172 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:34.009930 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:34.009935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:34.010213 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 18:44:34.010387 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:44:34.010644 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":147,"threads_started":1}
I20260927 18:44:34.012682 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:33.992996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:37233 (local address 127.18.235.190:45999)
0927 18:44:33.993696 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:33.993712 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:33.994115 (+ 403us) server_negotiation.cc:407] Connection header received
0927 18:44:33.995361 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:33.995375 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:33.995776 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:33.996119 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:33.997743 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:33.999738 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.001356 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.002065 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.002212 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.006168 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:34.006174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:34.006189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:34.006218 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:34.008098 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.008704 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.008710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.008716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.008854 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.009279 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.009283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.009285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.009739 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:34.009865 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:34.011863 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0927 18:44:34.012071 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":100,"threads_started":1}
I20260927 18:44:34.014966 32323 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45999
I20260927 18:44:34.015197 32323 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.015753 32323 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
W20260927 18:44:34.017886 32323 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:45999 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:44:34.018031 32323 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.018193 32323 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:34.018941 32323 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.019091 32323 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
W20260927 18:44:34.019733 32323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:44:34.145860 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:34.146170 32327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:34.146245 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:34.152012 32327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:34.152092 32327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:34.152191 32327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:44:34.158923 32327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:38095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=38849
--tserver_master_addrs=127.18.235.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:34.159866 32327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:34.160831 32327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:34.170413 32335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:34.170593 32333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:34.170432 32332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:34.171365 32327 server_base.cc:1034] running on GCE node
I20260927 18:44:34.171692 32327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:34.172274 32327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:34.173442 32327 hybrid_clock.cc:648] HybridClock initialized: now 1790534674173414 us; error 41 us; skew 500 ppm
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:34.176971 32327 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:44:34.178001 32327 webserver.cc:460] Webserver started at http://127.18.235.130:38849/ using document root <none> and password file <none>
I20260927 18:44:34.178514 32327 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:34.178587 32327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:34.203764 32327 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 18:44:34.206507 32343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.207760 32327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:34.207891 32327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "f0c4f6e05a090178e1976239f4a019c6"
server_key_iv: "b3c633863b45471a0d6fc42b53fb818d"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:34.208447 32327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:34.227922 32327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:34.230782 32327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:34.230954 32327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:34.231496 32327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:34.232340 32327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:34.232388 32327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.232439 32327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:34.232465 32327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.242813 32327 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:38095
I20260927 18:44:34.244056 32327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:34.245422 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 32327
I20260927 18:44:34.245858 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.235.148:42583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:37673
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:45999
--webserver_port=41167
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:34.249632 32456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:38095 every 8 connection(s)
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:34.270018 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:34.255498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:46173 (local address 127.18.235.190:45999)
0927 18:44:34.255644 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:34.255649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:34.257668 (+ 2019us) server_negotiation.cc:407] Connection header received
0927 18:44:34.258781 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:34.258785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:34.258845 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:34.258901 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:34.260221 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.261057 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.262787 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.263022 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.263111 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.265227 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:34.265230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:34.265232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:34.265255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:34.266470 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.266904 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.266907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.266908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.266984 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.267272 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.267275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.267277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.267439 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:34.267527 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:34.269714 (+ 2187us) server_negotiation.cc:299] Negotiation successful
0927 18:44:34.269842 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:44:34.272882 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:34.256033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45999 (local address 127.18.235.130:46173)
0927 18:44:34.257522 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:34.257568 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:34.258531 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:34.259301 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:34.259310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:34.259620 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:34.260078 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:34.260091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.261816 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.261819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:34.262665 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:34.262674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.263158 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.263162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:34.263328 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.263832 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:34.263848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:34.265057 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:34.266571 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:34.266577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:34.266590 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:34.266816 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:34.267062 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:34.267065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:34.267067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:34.267191 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:34.268676 (+ 1485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:34.268682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:34.269435 (+ 753us) client_negotiation.cc:769] Sending connection context
0927 18:44:34.270445 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0927 18:44:34.271941 (+ 1496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"thread_start_us":136,"threads_started":1}
I20260927 18:44:34.274468 32457 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45999
I20260927 18:44:34.274708 32457 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.275255 32457 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
W20260927 18:44:34.276866 32457 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:45999 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:44:34.276995 32457 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.277191 32457 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:34.277978 32457 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.278139 32457 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
W20260927 18:44:34.278908 32457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:44:34.388204 32459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:34.388586 32459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:34.388711 32459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:34.394490 32459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:34.394558 32459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:34.394637 32459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:44:34.400600 32459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.235.148:42583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:37673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=41167
--tserver_master_addrs=127.18.235.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:34.401595 32459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:34.402499 32459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:34.411123 32467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:34.411123 32466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:34.412253 32459 server_base.cc:1034] running on GCE node
W20260927 18:44:34.413223 32469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:34.413578 32459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:34.414170 32459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:34.415334 32459 hybrid_clock.cc:648] HybridClock initialized: now 1790534674415305 us; error 45 us; skew 500 ppm
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:34.418622 32459 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:44:34.419523 32459 webserver.cc:460] Webserver started at http://127.18.235.131:41167/ using document root <none> and password file <none>
I20260927 18:44:34.420121 32459 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:34.420208 32459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:34.447232 32459 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:44:34.450309 32477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.451830 32459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:34.451969 32459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "ef4cf75a54334915bdba42e78573b060"
format_stamp: "Formatted at 2026-09-27 18:44:33 on dist-test-slave-6lr6"
server_key: "a37e09ee3f59ea89f18b836eecf9881e"
server_key_iv: "3bc7f06b359549edc1b0638e7f462dfa"
server_key_version: "kuduclusterkey@0"
I20260927 18:44:34.452637 32459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:34.481873 32459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:34.485137 32459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:34.485314 32459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:34.485975 32459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:34.486910 32459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:34.486986 32459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.487041 32459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:34.487073 32459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.497718 32459 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:37673
I20260927 18:44:34.497731 32590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:37673 every 8 connection(s)
I20260927 18:44:34.498865 32459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:34.501583 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 32459
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:34.512646 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:34.502311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.18.235.190:45999)
0927 18:44:34.502553 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:34.502557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:34.502908 (+ 351us) server_negotiation.cc:407] Connection header received
0927 18:44:34.502965 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:34.502968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:34.503016 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:34.503091 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:34.504035 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.504803 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.506016 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.506324 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.506409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.508028 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:34.508031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:34.508034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:34.508053 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:34.509861 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.510281 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.510285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.510286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.510366 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.510722 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.510725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.510726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.511242 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:34.511643 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:34.512365 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 18:44:34.512476 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260927 18:44:34.514889 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:34.502078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:56249 (local address 127.18.235.190:45999)
0927 18:44:34.502200 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:34.502204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:34.502690 (+ 486us) server_negotiation.cc:407] Connection header received
0927 18:44:34.503836 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:34.503840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:34.503882 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:34.503929 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:34.505375 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.506070 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.507607 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.507923 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.507992 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.510572 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:34.510576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:34.510578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:34.510601 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:34.512308 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.513143 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.513146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.513148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.513225 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.513769 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.513772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.513774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.513959 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:34.514050 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:34.514659 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 18:44:34.514758 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:44:34.515879 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:34.501051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45999 (local address 127.18.235.131:56249)
0927 18:44:34.502551 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:34.502603 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:34.503615 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:34.504291 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:34.504300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:34.504599 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:34.505222 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:34.505240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.506196 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.506199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:34.507469 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:34.507480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.508159 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.508163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:34.508347 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.508826 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:34.508846 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:34.510409 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:34.512457 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:34.512463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:34.512475 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:34.513007 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:34.513501 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:34.513503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:34.513505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:34.513669 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:34.514127 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:34.514133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:34.514389 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 18:44:34.514591 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 18:44:34.514855 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":101,"threads_started":1}
I20260927 18:44:34.516803 32591 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45999
I20260927 18:44:34.517009 32591 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.517547 32591 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
W20260927 18:44:34.518847 32591 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:45999 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:44:34.518947 32591 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.519088 32591 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:34.519943 32591 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:34.520095 32591 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
W20260927 18:44:34.521286 32591 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:44:34 dist-test-slave-6lr6 krb5kdc[31129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:35.019951 32323 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:35.020247 32323 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:35.021399 32125 ts_manager.cc:194] Registered new tserver with Master: c8100190a1204be780ac386ca6746f3e (127.18.235.129:33545)
I20260927 18:44:35.023046 32125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:37233
Sep 27 18:44:35 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:35.279126 32457 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:35.279392 32457 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:35.280064 32125 ts_manager.cc:194] Registered new tserver with Master: 63e94e23c7ad42d38d5b7b2a4fa9ae67 (127.18.235.130:38095)
I20260927 18:44:35.280755 32125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:46173
Sep 27 18:44:35 dist-test-slave-6lr6 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for HTTP/127.18.235.148@KRBTEST.COM
I20260927 18:44:35.521473 32591 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:35.521746 32591 heartbeater.cc:511] Master 127.18.235.190:45999 requested a full tablet report, sending...
I20260927 18:44:35.529392 32125 ts_manager.cc:194] Registered new tserver with Master: ef4cf75a54334915bdba42e78573b060 (127.18.235.131:37673)
I20260927 18:44:35.530186 32125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:56249
I20260927 18:44:35.538048 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:44:35.555253 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.541166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.18.235.190:45999)
0927 18:44:35.541911 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.541916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.541929 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:44:35.541965 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.541967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.542011 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.542894 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:35.542982 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.543813 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.544995 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.545382 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.546214 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.550023 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:35.550027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:35.550030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:35.550056 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:35.551792 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:35.552276 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:35.552279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:35.552281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:35.552363 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:35.553881 (+ 1518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:35.553884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:35.553886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:35.554096 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:35.554893 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.554965 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.555084 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":90,"threads_started":1}
I20260927 18:44:35.559278 32125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:35.563751 32125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:35.590067 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.575738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:37673 (local address 127.0.0.1:51248)
0927 18:44:35.577848 (+ 2110us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.577890 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.578005 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.578710 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.578716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.578738 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:35.578856 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.578862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.583139 (+ 4277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.583143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.585210 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.585220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.588192 (+ 2972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.588196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.588307 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.589696 (+ 1389us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.589787 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.589903 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1946,"thread_start_us":79,"threads_started":1}
I20260927 18:44:35.590716 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.576477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.18.235.131:37673)
0927 18:44:35.576809 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.576816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.578144 (+ 1328us) server_negotiation.cc:407] Connection header received
0927 18:44:35.578222 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.578227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.578421 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.578547 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:35.578974 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.583033 (+ 4059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.587111 (+ 4078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.587923 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.588049 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.590441 (+ 2392us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.590492 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.590583 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260927 18:44:35.591933 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.575740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33545 (local address 127.0.0.1:50484)
0927 18:44:35.580836 (+ 5096us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.580848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.580927 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.581540 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.581543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.581559 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:35.581638 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.581643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.585324 (+ 3681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.585327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.586483 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.586494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.591556 (+ 5062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.591559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.591645 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.591668 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.591732 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.591795 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5039,"mutex_wait_us":481,"thread_start_us":90,"threads_started":1}
I20260927 18:44:35.592308 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.575964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.18.235.129:33545)
0927 18:44:35.576970 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.576975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.581026 (+ 4051us) server_negotiation.cc:407] Connection header received
0927 18:44:35.581121 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.581126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.581299 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.581418 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:35.581744 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.582868 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.586618 (+ 3750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.589434 (+ 2816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.589532 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.589659 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.592097 (+ 2438us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.592186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":120,"threads_started":1}
I20260927 18:44:35.593796 32525 tablet_service.cc:1467] Processing CreateTablet for tablet 77cb3d60c79444299951d5ea8756b302 (DEFAULT_TABLE table=test-table [id=cca18fcdd3e94331b74e8b1b7ee98985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:35.594892 32525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cb3d60c79444299951d5ea8756b302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:35.595260 32257 tablet_service.cc:1467] Processing CreateTablet for tablet 77cb3d60c79444299951d5ea8756b302 (DEFAULT_TABLE table=test-table [id=cca18fcdd3e94331b74e8b1b7ee98985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:35.596275 32257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cb3d60c79444299951d5ea8756b302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:35.597728 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.576567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38095 (local address 127.0.0.1:55328)
0927 18:44:35.589249 (+ 12682us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.589262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.589346 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.593060 (+ 3714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.593063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.593080 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:35.593178 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.593184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.595313 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.595316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.596373 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.596383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.597387 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.597391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.597479 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.597502 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.597548 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.597608 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12621,"thread_start_us":81,"threads_started":1}
I20260927 18:44:35.598284 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.576943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.18.235.130:38095)
0927 18:44:35.592629 (+ 15686us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.592636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.592660 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:44:35.592730 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.592735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.592907 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.593321 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:35.593455 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.595188 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.596516 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.597261 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.597898 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.598021 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.598065 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.598150 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15582,"thread_start_us":80,"threads_started":1}
I20260927 18:44:35.600890 32375 tablet_service.cc:1467] Processing CreateTablet for tablet 77cb3d60c79444299951d5ea8756b302 (DEFAULT_TABLE table=test-table [id=cca18fcdd3e94331b74e8b1b7ee98985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:35.601846 32375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cb3d60c79444299951d5ea8756b302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:35.610468 32628 tablet_bootstrap.cc:492] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: Bootstrap starting.
I20260927 18:44:35.612659 32628 tablet_bootstrap.cc:654] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:35.613030 32627 tablet_bootstrap.cc:492] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: Bootstrap starting.
I20260927 18:44:35.613464 32628 log.cc:824] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:35.615586 32628 tablet_bootstrap.cc:492] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: No bootstrap required, opened a new log
I20260927 18:44:35.615782 32628 ts_tablet_manager.cc:1397] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:35.616496 32627 tablet_bootstrap.cc:654] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:35.617398 32626 tablet_bootstrap.cc:492] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Bootstrap starting.
I20260927 18:44:35.618305 32627 log.cc:824] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:35.619331 32626 tablet_bootstrap.cc:654] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:35.620071 32626 log.cc:824] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:35.620561 32628 raft_consensus.cc:348] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.620812 32628 raft_consensus.cc:374] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:35.620862 32628 raft_consensus.cc:729] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e94e23c7ad42d38d5b7b2a4fa9ae67, State: Initialized, Role: FOLLOWER
I20260927 18:44:35.621359 32628 consensus_queue.cc:260] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.622354 32628 ts_tablet_manager.cc:1428] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 18:44:35.622687 32457 heartbeater.cc:503] Master 127.18.235.190:45999 was elected leader, sending a full tablet report...
I20260927 18:44:35.627300 32626 tablet_bootstrap.cc:492] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: No bootstrap required, opened a new log
I20260927 18:44:35.627494 32626 ts_tablet_manager.cc:1397] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260927 18:44:35.629494 32627 tablet_bootstrap.cc:492] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: No bootstrap required, opened a new log
I20260927 18:44:35.629694 32627 ts_tablet_manager.cc:1397] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20260927 18:44:35.630539 32626 raft_consensus.cc:348] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.630788 32626 raft_consensus.cc:374] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:35.630847 32626 raft_consensus.cc:729] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4cf75a54334915bdba42e78573b060, State: Initialized, Role: FOLLOWER
I20260927 18:44:35.631321 32626 consensus_queue.cc:260] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.632282 32626 ts_tablet_manager.cc:1428] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:44:35.632643 32591 heartbeater.cc:503] Master 127.18.235.190:45999 was elected leader, sending a full tablet report...
I20260927 18:44:35.633565 32627 raft_consensus.cc:348] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.633817 32627 raft_consensus.cc:374] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:35.633882 32627 raft_consensus.cc:729] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8100190a1204be780ac386ca6746f3e, State: Initialized, Role: FOLLOWER
I20260927 18:44:35.634356 32627 consensus_queue.cc:260] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.635006 32323 heartbeater.cc:503] Master 127.18.235.190:45999 was elected leader, sending a full tablet report...
I20260927 18:44:35.635330 32627 ts_tablet_manager.cc:1428] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:44:35.638387 32635 raft_consensus.cc:482] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:35.638578 32635 raft_consensus.cc:504] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.641927 32635 leader_election.cc:288] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8100190a1204be780ac386ca6746f3e (127.18.235.129:33545), 63e94e23c7ad42d38d5b7b2a4fa9ae67 (127.18.235.130:38095)
I20260927 18:44:35.650122 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.641781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:43763 (local address 127.18.235.130:38095)
0927 18:44:35.641896 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.641900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.642802 (+ 902us) server_negotiation.cc:407] Connection header received
0927 18:44:35.643344 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.643348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.643391 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.643437 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:35.644059 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.644819 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.646401 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.647025 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.647158 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.647228 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.649899 (+ 2671us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.649984 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:44:35.650483 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.641433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38095 (local address 127.18.235.131:43763)
0927 18:44:35.642727 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.642739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.642976 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.643827 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.643830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.643843 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:35.643938 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.643944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.644960 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.644963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.646266 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.646278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.649636 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.649640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.649742 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.649768 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.650251 (+ 483us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.650329 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"thread_start_us":83,"threads_started":1}
I20260927 18:44:35.651212 32396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb3d60c79444299951d5ea8756b302" candidate_uuid: "ef4cf75a54334915bdba42e78573b060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" is_pre_election: true
I20260927 18:44:35.651490 32396 raft_consensus.cc:2454] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef4cf75a54334915bdba42e78573b060 in term 0.
I20260927 18:44:35.651970 32479 leader_election.cc:302] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63e94e23c7ad42d38d5b7b2a4fa9ae67, ef4cf75a54334915bdba42e78573b060; no voters:
I20260927 18:44:35.652100 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.640386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:54899 (local address 127.18.235.129:33545)
0927 18:44:35.640619 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.640624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.642605 (+ 1981us) server_negotiation.cc:407] Connection header received
0927 18:44:35.643101 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.643104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.643152 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.643201 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:35.646431 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.647157 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.649607 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.650221 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.650334 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.650376 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.651910 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.651984 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 18:44:35.652453 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.639899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33545 (local address 127.18.235.131:54899)
0927 18:44:35.642515 (+ 2616us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.642530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.642872 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.643565 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.643568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.643585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:35.643713 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.643719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.647298 (+ 3579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.647301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.648451 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.648463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.650839 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.650843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.651764 (+ 921us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.651789 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.652252 (+ 463us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.652323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2551,"thread_start_us":105,"threads_started":1}
I20260927 18:44:35.652683 32633 raft_consensus.cc:2781] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:35.652812 32633 raft_consensus.cc:482] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:35.652900 32633 raft_consensus.cc:3037] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:35.652925 32259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb3d60c79444299951d5ea8756b302" candidate_uuid: "ef4cf75a54334915bdba42e78573b060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8100190a1204be780ac386ca6746f3e" is_pre_election: true
I20260927 18:44:35.653205 32259 raft_consensus.cc:2454] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef4cf75a54334915bdba42e78573b060 in term 0.
I20260927 18:44:35.654361 32633 raft_consensus.cc:504] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.654690 32633 leader_election.cc:288] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [CANDIDATE]: Term 1 election: Requested vote from peers c8100190a1204be780ac386ca6746f3e (127.18.235.129:33545), 63e94e23c7ad42d38d5b7b2a4fa9ae67 (127.18.235.130:38095)
I20260927 18:44:35.655048 32396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb3d60c79444299951d5ea8756b302" candidate_uuid: "ef4cf75a54334915bdba42e78573b060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67"
I20260927 18:44:35.655102 32259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cb3d60c79444299951d5ea8756b302" candidate_uuid: "ef4cf75a54334915bdba42e78573b060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8100190a1204be780ac386ca6746f3e"
I20260927 18:44:35.655169 32396 raft_consensus.cc:3037] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:35.655193 32259 raft_consensus.cc:3037] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:35.656428 32396 raft_consensus.cc:2454] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef4cf75a54334915bdba42e78573b060 in term 1.
I20260927 18:44:35.656431 32259 raft_consensus.cc:2454] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef4cf75a54334915bdba42e78573b060 in term 1.
I20260927 18:44:35.656762 32479 leader_election.cc:302] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63e94e23c7ad42d38d5b7b2a4fa9ae67, ef4cf75a54334915bdba42e78573b060; no voters:
I20260927 18:44:35.656905 32633 raft_consensus.cc:2781] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:35.656981 32633 raft_consensus.cc:686] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 1 LEADER]: Becoming Leader. State: Replica: ef4cf75a54334915bdba42e78573b060, State: Running, Role: LEADER
I20260927 18:44:35.657308 32633 consensus_queue.cc:237] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } }
I20260927 18:44:35.660785 32125 catalog_manager.cc:5495] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef4cf75a54334915bdba42e78573b060 (127.18.235.131). New cstate: current_term: 1 leader_uuid: "ef4cf75a54334915bdba42e78573b060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef4cf75a54334915bdba42e78573b060" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 37673 } health_report { overall_health: HEALTHY } } }
I20260927 18:44:35.686335 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.681858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.18.235.131:37673)
0927 18:44:35.681981 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.681986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.681999 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:44:35.682076 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.682079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.682134 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.682234 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:35.682491 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.683274 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.684524 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.684874 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.684975 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.685406 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:35.685906 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:35.686030 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.686128 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.686186 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:44:35.693041 32259 raft_consensus.cc:1261] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e [term 1 FOLLOWER]: Refusing update from remote peer ef4cf75a54334915bdba42e78573b060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:35.693040 32396 raft_consensus.cc:1261] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 1 FOLLOWER]: Refusing update from remote peer ef4cf75a54334915bdba42e78573b060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:35.693786 32633 consensus_queue.cc:1035] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8100190a1204be780ac386ca6746f3e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:35.693945 32635 consensus_queue.cc:1035] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e94e23c7ad42d38d5b7b2a4fa9ae67" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:35.702194 32640 mvcc.cc:205] Tried to move back new op lower bound from 7334030031630745600 to 7334030031495819264. Current Snapshot: MvccSnapshot[applied={T|T < 7334030031630745600}]
I20260927 18:44:35.704650 32641 mvcc.cc:205] Tried to move back new op lower bound from 7334030031630745600 to 7334030031495819264. Current Snapshot: MvccSnapshot[applied={T|T < 7334030031630745600}]
I20260927 18:44:35.715631 32642 mvcc.cc:205] Tried to move back new op lower bound from 7334030031630745600 to 7334030031495819264. Current Snapshot: MvccSnapshot[applied={T|T < 7334030031630745600 or (T in {7334030031630745600})}]
I20260927 18:44:35.716400 32125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:35.716559 32125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:35.718966 32125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f87978cb6c44ea59c1fd86d622d96ce: Sending DeleteTablet for 3 replicas of tablet 77cb3d60c79444299951d5ea8756b302
I20260927 18:44:35.720183 32257 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cb3d60c79444299951d5ea8756b302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:35 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:50484
I20260927 18:44:35.720630 32375 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cb3d60c79444299951d5ea8756b302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:35 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:55328
I20260927 18:44:35.721066 32525 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cb3d60c79444299951d5ea8756b302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:35 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:51248
I20260927 18:44:35.721871 32649 tablet_replica.cc:330] T 77cb3d60c79444299951d5ea8756b302 P c8100190a1204be780ac386ca6746f3e: stopping tablet replica
I20260927 18:44:35.722246 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 32163
I20260927 18:44:35.723704 32651 tablet_replica.cc:330] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: stopping tablet replica
I20260927 18:44:35.723965 32651 raft_consensus.cc:2229] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:35.724324 32651 raft_consensus.cc:2258] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:35.724644 32650 tablet_replica.cc:330] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: stopping tablet replica
I20260927 18:44:35.724893 32650 raft_consensus.cc:2229] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:35.725214 32650 raft_consensus.cc:2258] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:35.730325 32650 ts_tablet_manager.cc:1905] T 77cb3d60c79444299951d5ea8756b302 P 63e94e23c7ad42d38d5b7b2a4fa9ae67: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:44:35.730749 32110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:33545 (error 108)
I20260927 18:44:35.731451 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 32327
I20260927 18:44:35.732118 32651 ts_tablet_manager.cc:1905] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:35.734714 32651 ts_tablet_manager.cc:1918] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:35.734800 32651 log.cc:1196] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/77cb3d60c79444299951d5ea8756b302
I20260927 18:44:35.734867 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.734580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33545 (local address 127.0.0.1:50500)
0927 18:44:35.734706 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.734784 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:33545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:44:35.735116 32651 ts_tablet_manager.cc:1939] T 77cb3d60c79444299951d5ea8756b302 P ef4cf75a54334915bdba42e78573b060: Deleting consensus metadata
W20260927 18:44:35.735168 32110 catalog_manager.cc:4584] TS c8100190a1204be780ac386ca6746f3e (127.18.235.129:33545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77cb3d60c79444299951d5ea8756b302: Network error: Client connection negotiation failed: client connection to 127.18.235.129:33545: connect: Connection refused (error 111)
I20260927 18:44:35.736164 32109 catalog_manager.cc:4841] TS ef4cf75a54334915bdba42e78573b060 (127.18.235.131:37673): tablet 77cb3d60c79444299951d5ea8756b302 (table test-table [id=cca18fcdd3e94331b74e8b1b7ee98985]) successfully deleted
I20260927 18:44:35.738766 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 32459
I20260927 18:44:35.742180 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.741805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38095 (local address 127.0.0.1:55338)
0927 18:44:35.742019 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.742082 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.130:38095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 18:44:35.747128 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 32092
I20260927 18:44:35.752373 19374 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:44:36.085461 19374 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:44:36.085565 19374 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:44:36.418534 19374 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:44:36.418 UTC [31159] LOG: received smart shutdown request
2026-09-27 18:44:36.422 UTC [31159] LOG: background worker "logical replication launcher" (PID 31168) exited with exit code 1
2026-09-27 18:44:36.422 UTC [31163] LOG: shutting down
2026-09-27 18:44:36.555 UTC [31159] LOG: database system is shut down
2026-09-27T18:44:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44778 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:36 dist-test-slave-6lr6 krb5kdc[32663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:36 dist-test-slave-6lr6 krb5kdc[32663](info): set up 2 sockets
Sep 27 18:44:36 dist-test-slave-6lr6 krb5kdc[32663](info): commencing operation
krb5kdc: starting...
W20260927 18:44:37.592800 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:44:37 dist-test-slave-6lr6 krb5kdc[32663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:44:37.697564 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35741
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35741,127.18.235.189:37555,127.18.235.188:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:37.797637 32676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:37.797848 32676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:37.797884 32676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:37.801643 32676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:37.801682 32676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:37.801699 32676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:37.801713 32676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:37.801728 32676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:37.806528 32676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35741,127.18.235.189:37555,127.18.235.188:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:37.807335 32676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:37.808171 32676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:37.815502 32682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:37.815523 32684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:37.815548 32681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:37.815650 32676 server_base.cc:1034] running on GCE node
I20260927 18:44:37.816198 32676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:37.817169 32676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:37.818313 32676 hybrid_clock.cc:648] HybridClock initialized: now 1790534677818294 us; error 33 us; skew 500 ppm
Sep 27 18:44:37 dist-test-slave-6lr6 krb5kdc[32663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:37.820734 32676 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:37.821507 32676 webserver.cc:460] Webserver started at http://127.18.235.190:44673/ using document root <none> and password file <none>
I20260927 18:44:37.821957 32676 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:37.822007 32676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:37.822165 32676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:37.823899 32676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "a70e3673be9b4d95b444c63f13a1aca4"
format_stamp: "Formatted at 2026-09-27 18:44:37 on dist-test-slave-6lr6"
server_key: "aacbc12bbfbf1f64958ef3855ab7f53a"
server_key_iv: "7b92345c4ba5ee2702e0fcf13408b00a"
server_key_version: "encryptionkey@0"
I20260927 18:44:37.824358 32676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "a70e3673be9b4d95b444c63f13a1aca4"
format_stamp: "Formatted at 2026-09-27 18:44:37 on dist-test-slave-6lr6"
server_key: "aacbc12bbfbf1f64958ef3855ab7f53a"
server_key_iv: "7b92345c4ba5ee2702e0fcf13408b00a"
server_key_version: "encryptionkey@0"
I20260927 18:44:37.827575 32676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:44:37.829639 32691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:37.830677 32676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:37.830778 32676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "a70e3673be9b4d95b444c63f13a1aca4"
format_stamp: "Formatted at 2026-09-27 18:44:37 on dist-test-slave-6lr6"
server_key: "aacbc12bbfbf1f64958ef3855ab7f53a"
server_key_iv: "7b92345c4ba5ee2702e0fcf13408b00a"
server_key_version: "encryptionkey@0"
I20260927 18:44:37.830859 32676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:37.857788 32676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:37.860473 32676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:37.860641 32676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:37.866750 32676 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:35741
I20260927 18:44:37.867005 32743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:35741 every 8 connection(s)
I20260927 18:44:37.867568 32676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:37.870038 32744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:37.871749 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 32676
I20260927 18:44:37.871863 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:44:37.872107 19374 external_mini_cluster.cc:1428] Setting key 80e1eb019595354ebfa4d9af709ddf10
I20260927 18:44:37.873529 32744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.189@KRBTEST.COM" created.
Sep 27 18:44:37 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:37.887131 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:37.875529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35741 (local address 127.0.0.1:42846)
0927 18:44:37.876096 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:37.876138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:37.876879 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:37.877601 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:37.877605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:37.877879 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:37.878132 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:37.878143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:37.879626 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:37.879629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:37.880337 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:37.880345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:37.880899 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:37.880902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:37.880994 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:37.881419 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:37.881433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:37.883007 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:37.884559 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:37.884566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:37.884580 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:37.884780 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:37.885041 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:37.885044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:37.885046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:37.885187 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:37.885677 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:37.885683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:37.885799 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 18:44:37.885930 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:44:37.886222 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":3,"spinlock_wait_cycles":50560,"thread_start_us":139,"threads_started":1}
I20260927 18:44:37.887131 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:37.875529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.18.235.190:35741)
0927 18:44:37.875965 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:37.875982 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:37.876239 (+ 257us) server_negotiation.cc:407] Connection header received
0927 18:44:37.877191 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:37.877197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:37.877424 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:37.877615 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:37.878267 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:37.879527 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:37.880439 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:37.880783 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:37.880896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:37.883127 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:37.883133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:37.883141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:37.883160 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:37.884457 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:37.884870 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:37.884874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:37.884876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:37.884964 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:37.885263 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:37.885266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:37.885268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:37.885581 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:37.885689 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:37.885999 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 18:44:37.886279 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":292,"thread_start_us":135,"threads_started":1}
I20260927 18:44:37.889093 32744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:37.889732 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:37.889478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58036)
0927 18:44:37.889591 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:37.889674 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:44:37.889894 32693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
I20260927 18:44:37.890580 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:37.890399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58050)
0927 18:44:37.890486 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:37.890530 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:44:37.890692 32744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:37555: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab.
Entry for principal kudu/127.18.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab.
I20260927 18:44:37.894004 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:37555
--webserver_interface=127.18.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab
--principal=kudu/127.18.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35741,127.18.235.189:37555,127.18.235.188:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:37.939939 32744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } attempt: 1
I20260927 18:44:37.940528 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:37.940302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58064)
0927 18:44:37.940419 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:37.940467 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 18:44:37.941175 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:37.940962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58068)
0927 18:44:37.941050 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:37.941090 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:44:37.941280 32744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:37555: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
W20260927 18:44:37.995142 32751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:37.995355 32751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:37.995400 32751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:37.999410 32751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:37.999462 32751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:37.999500 32751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:37.999518 32751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:37.999541 32751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:38.004374 32751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35741,127.18.235.189:37555,127.18.235.188:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab
--principal=kudu/127.18.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:38.005209 32751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:38.006054 32751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:44:38.009436 32744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } attempt: 2
I20260927 18:44:38.010234 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.009931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58080)
0927 18:44:38.010087 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.010153 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:44:38.010947 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.010731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58092)
0927 18:44:38.010833 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.010885 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:44:38.011071 32744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:37555: Network error: Client connection negotiation failed: client connection to 127.18.235.189:37555: connect: Connection refused (error 111)
W20260927 18:44:38.013876 32759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:38.013911 32757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:38.013985 32751 server_base.cc:1034] running on GCE node
W20260927 18:44:38.013893 32756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:38.014591 32751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:38.015502 32751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:38.016649 32751 hybrid_clock.cc:648] HybridClock initialized: now 1790534678016634 us; error 29 us; skew 500 ppm
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:38.019133 32751 init.cc:377] Logged in from keytab as kudu/127.18.235.189@KRBTEST.COM (short username kudu)
I20260927 18:44:38.019846 32751 webserver.cc:460] Webserver started at http://127.18.235.189:45997/ using document root <none> and password file <none>
I20260927 18:44:38.020287 32751 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:38.020337 32751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:38.020493 32751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:38.022296 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data/instance:
uuid: "8bd9df0db252450dbf181086e14fc3ea"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-6lr6"
server_key: "aeab6ec0ce48504bce701b4daf61c47c"
server_key_iv: "dc8bf53c220473f06ffc9b2e673c7519"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.022760 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal/instance:
uuid: "8bd9df0db252450dbf181086e14fc3ea"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-6lr6"
server_key: "aeab6ec0ce48504bce701b4daf61c47c"
server_key_iv: "dc8bf53c220473f06ffc9b2e673c7519"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.026084 32751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:44:38.028800 32766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:38.030251 32751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:44:38.030366 32751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal
uuid: "8bd9df0db252450dbf181086e14fc3ea"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-6lr6"
server_key: "aeab6ec0ce48504bce701b4daf61c47c"
server_key_iv: "dc8bf53c220473f06ffc9b2e673c7519"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.030465 32751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:38.051818 32751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:38.054216 32751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:38.054360 32751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:38.060379 32751 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.189:37555
I20260927 18:44:38.060603 350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.189:37555 every 8 connection(s)
I20260927 18:44:38.061239 32751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
I20260927 18:44:38.063656 351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:38.067018 351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:38.068742 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 32751
I20260927 18:44:38.068853 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-1/wal/instance
I20260927 18:44:38.069082 19374 external_mini_cluster.cc:1428] Setting key 848144eae4627a61e45a3167854bee56
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.188@KRBTEST.COM" created.
I20260927 18:44:38.080190 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.068646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.18.235.190:35741)
0927 18:44:38.068777 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.068782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.069620 (+ 838us) server_negotiation.cc:407] Connection header received
0927 18:44:38.070464 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.070468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.070533 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.070615 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.071679 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.072196 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.073770 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.073995 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.074074 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.076547 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.076549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.076551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.076575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.078354 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.078771 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.078774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.078776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.078849 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.079198 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.079202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.079204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.079380 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.079465 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.079913 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.080034 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:44:38.080821 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.069017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35741 (local address 127.0.0.1:42860)
0927 18:44:38.069465 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.069513 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.070280 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.070737 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.070743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.071025 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.071542 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.071559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.072308 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.072312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.073660 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.073672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.074138 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.074142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.074334 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.074819 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.074837 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.076387 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.078461 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.078466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.078476 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.078694 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.078930 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.078933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.078935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.079103 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.079474 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.079480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.079692 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.079842 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.080133 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260927 18:44:38.082043 351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:44:38.090946 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.082448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58102)
0927 18:44:38.082776 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.082786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.082842 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.083203 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.083205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.083316 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.083393 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.083397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.084715 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.084717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.085279 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.085286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.085981 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.085984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.086048 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.086420 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.086440 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.088256 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.089772 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.089775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.089776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.089940 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.090158 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.090160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.090161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.090220 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.090592 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.090594 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.090663 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.090739 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.090822 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"spinlock_wait_cycles":1920}
I20260927 18:44:38.091112 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.082550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.18.235.189:37555)
0927 18:44:38.082821 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.082832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.082854 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:44:38.082898 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.082904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.083097 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.083233 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.083485 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.084617 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.085373 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.085886 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.086009 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.088361 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.088369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.088378 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.088397 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.089691 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.090019 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.090023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.090025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.090094 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.090282 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.090285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.090286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.090503 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.090595 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.090820 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.090926 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":3328,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab.
Entry for principal kudu/127.18.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab.
I20260927 18:44:38.092494 351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:38.092898 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:36261
--webserver_interface=127.18.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab
--principal=kudu/127.18.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35741,127.18.235.189:37555,127.18.235.188:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:38.093068 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.092849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48314)
0927 18:44:38.092944 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.093011 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 18:44:38.093284 301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
I20260927 18:44:38.093976 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.093800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48330)
0927 18:44:38.093887 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.093931 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:44:38.094100 351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:36261: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
I20260927 18:44:38.102198 32744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } attempt: 3
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:44:38.109247 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.102614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.18.235.189:37555)
0927 18:44:38.102713 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.102717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.102762 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:44:38.102882 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.102885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.102922 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.102978 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.103356 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.103875 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.104649 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.104916 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.104994 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.106719 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.106722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.106725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.106747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.107894 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.108250 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.108254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.108255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.108321 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.108546 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.108549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.108551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.108698 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.108775 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.109002 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.109131 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:44:38.109242 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.102583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58108)
0927 18:44:38.102701 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.102714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.102806 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.103005 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.103008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.103163 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.103252 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.103257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.103973 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.103977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.104543 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.104552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.105019 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.105022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.105148 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.105522 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.105537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.106605 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.107989 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.107992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.107994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.108170 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.108397 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.108400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.108402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.108474 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.108779 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.108783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.108866 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.108947 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.109065 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 18:44:38.110317 32744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:38.110890 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.110697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48334)
0927 18:44:38.110795 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.110837 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 18:44:38.111459 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.111254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48338)
0927 18:44:38.111372 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.111414 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 18:44:38.111567 32744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:36261: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
I20260927 18:44:38.142705 32744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } attempt: 1
I20260927 18:44:38.143291 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.143062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48346)
0927 18:44:38.143168 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.143213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 18:44:38.143308 351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } attempt: 1
I20260927 18:44:38.143847 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.143635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48348)
0927 18:44:38.143736 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.143781 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 18:44:38.143937 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.143743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48350)
0927 18:44:38.143838 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.143882 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 18:44:38.144061 32744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:36261: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
I20260927 18:44:38.144470 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.144295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48356)
0927 18:44:38.144386 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.144426 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:44:38.144569 351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:36261: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
W20260927 18:44:38.198681 358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:38.198951 358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:38.199016 358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:38.202979 358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:38.203037 358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:38.203068 358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:38.203104 358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:38.203136 358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:38.208212 358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35741,127.18.235.189:37555,127.18.235.188:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab
--principal=kudu/127.18.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:38.209218 358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:38.210191 358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:44:38.212707 351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } attempt: 2
I20260927 18:44:38.213495 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.213219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48370)
0927 18:44:38.213361 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.213421 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:44:38.214231 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.213979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48376)
0927 18:44:38.214101 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.214161 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:44:38.214418 351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:36261: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
W20260927 18:44:38.218031 363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:38.218093 366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:38.218191 358 server_base.cc:1034] running on GCE node
W20260927 18:44:38.218112 364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:38.218791 358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:38.219216 32744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } attempt: 2
I20260927 18:44:38.219841 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.219579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48392)
0927 18:44:38.219709 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.219764 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:44:38.219858 358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:38.220443 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.220254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48400)
0927 18:44:38.220348 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.220389 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:44:38.220543 32744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:36261: Network error: Client connection negotiation failed: client connection to 127.18.235.188:36261: connect: Connection refused (error 111)
I20260927 18:44:38.221014 358 hybrid_clock.cc:648] HybridClock initialized: now 1790534678220962 us; error 64 us; skew 500 ppm
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:38.223421 358 init.cc:377] Logged in from keytab as kudu/127.18.235.188@KRBTEST.COM (short username kudu)
I20260927 18:44:38.224164 358 webserver.cc:460] Webserver started at http://127.18.235.188:40095/ using document root <none> and password file <none>
I20260927 18:44:38.224619 358 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:38.224669 358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:38.224817 358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:38.226643 358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data/instance:
uuid: "460232dba1fb436f84ddf86e9f2cdfe5"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-6lr6"
server_key: "b780778862334527102f76ff490031f9"
server_key_iv: "069e3cd100a46d3cfeed61f8898d23b8"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.227162 358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal/instance:
uuid: "460232dba1fb436f84ddf86e9f2cdfe5"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-6lr6"
server_key: "b780778862334527102f76ff490031f9"
server_key_iv: "069e3cd100a46d3cfeed61f8898d23b8"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.230453 358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:44:38.232583 373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:38.233810 358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:38.233911 358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal
uuid: "460232dba1fb436f84ddf86e9f2cdfe5"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-6lr6"
server_key: "b780778862334527102f76ff490031f9"
server_key_iv: "069e3cd100a46d3cfeed61f8898d23b8"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.233992 358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:38.257248 358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:38.259743 358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:38.259882 358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:38.265846 358 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.188:36261
I20260927 18:44:38.266103 425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.188:36261 every 8 connection(s)
I20260927 18:44:38.266737 358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
I20260927 18:44:38.267701 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 358
I20260927 18:44:38.267803 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534532155427-19374-0/minicluster-data/master-2/wal/instance
I20260927 18:44:38.268042 19374 external_mini_cluster.cc:1428] Setting key 9daa5da248196f0d3a055cd5632a1bd3
I20260927 18:44:38.269624 426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:38.273918 426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:38.281540 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.269224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.18.235.188:36261)
0927 18:44:38.269605 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.269623 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.269664 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:44:38.270439 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.270460 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.270856 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.271197 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.271793 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.273241 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.274010 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.274667 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.274814 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.277242 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.277248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.277259 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.277288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.278865 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.279288 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.279292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.279296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.279397 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.279655 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.279659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.279661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.280005 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.280173 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.280444 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.280713 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260927 18:44:38.283902 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.275802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.18.235.190:35741)
0927 18:44:38.275908 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.275911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.276236 (+ 325us) server_negotiation.cc:407] Connection header received
0927 18:44:38.276351 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.276354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.276401 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.276464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.277121 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.277807 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.278699 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.279021 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.279096 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.281050 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.281054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.281056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.281083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.282334 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.282762 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.282765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.282767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.282853 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.283064 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.283067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.283069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.283275 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.283362 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.283656 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.283759 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:44:38.283937 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.275775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35741 (local address 127.0.0.1:42876)
0927 18:44:38.276124 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.276159 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.276266 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.276532 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.276538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.276873 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.276991 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.276996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.277914 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.277916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.278581 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.278591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.279132 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.279135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.279248 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.279672 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.279683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.280919 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.282428 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.282435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.282440 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.282679 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.282931 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.282932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.282934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.282998 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.283370 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.283375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.283472 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.283603 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.283759 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":131,"threads_started":1}
I20260927 18:44:38.284796 426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:44:38.291603 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.283789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.18.235.190:35741)
0927 18:44:38.284068 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.284071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.284084 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:44:38.284386 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.284389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.284430 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.284501 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.285027 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.285659 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.286450 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.286700 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.286769 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.288931 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.288935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.288937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.288961 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.290245 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.290598 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.290601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.290603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.290673 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.290896 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.290899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.290901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.291046 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.291223 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.291356 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.291451 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260927 18:44:38.291821 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.285220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:37555 (local address 127.0.0.1:58112)
0927 18:44:38.285350 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.285362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.285424 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.285648 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.285650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.285780 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.285875 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.285881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.286529 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.286532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.287161 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.287171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.287564 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.287568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.287666 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.287983 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.287997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.289145 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.290640 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.290643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.290645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.290815 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.291051 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.291054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.291056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.291128 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.291442 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.291445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.291528 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.291630 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.291724 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:44:38.291961 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.285332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.18.235.189:37555)
0927 18:44:38.285467 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.285472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.285484 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:44:38.285524 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.285527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.285567 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.285646 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.285973 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.286430 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.287258 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.287465 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.287537 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.289275 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.289279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.289282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.289307 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.290533 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.290897 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.290900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.290902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.290975 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.291202 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.291205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.291207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.291361 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.291440 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.291716 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.291817 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:44:38.292665 426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:44:38.299544 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.293157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48412)
0927 18:44:38.293321 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.293330 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.293386 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.293586 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.293587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.293705 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.293776 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.293782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.294417 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.294420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.294886 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.294893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.295294 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.295297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.295372 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.295756 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.295765 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.296755 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.298317 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.298319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.298321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.298494 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.298751 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.298754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.298756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.298834 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.299168 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.299171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.299258 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.299347 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.299445 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 18:44:38.299680 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.293235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.18.235.188:36261)
0927 18:44:38.293325 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.293329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.293363 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:44:38.293456 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.293459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.293509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.293582 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.293871 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.294323 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.294984 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.295200 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.295283 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.296875 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.296879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.296882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.296905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.298224 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.298580 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.298584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.298586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.298670 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.298914 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.298916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.298918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.299081 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.299172 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.299410 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.299511 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:44:38.300364 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.292777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.18.235.189:37555)
0927 18:44:38.292866 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.292869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.292923 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:44:38.293008 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.293011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.293046 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.293145 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.293787 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.294269 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.294964 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.295191 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.295266 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.297424 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.297428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.297430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.297455 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.298647 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.299036 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.299039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.299040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.299120 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.299345 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.299347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.299349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.299528 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.299744 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.300134 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.300235 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:44:38.303030 426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: Bootstrap starting.
I20260927 18:44:38.303233 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:38.305564 351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } attempt: 3
I20260927 18:44:38.305711 426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:38.306571 426 log.cc:824] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:44:38.312341 426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: No bootstrap required, opened a new log
I20260927 18:44:38.314572 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.305983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48428)
0927 18:44:38.306118 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.306130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.306193 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.306843 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.306846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.307010 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.307095 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.307102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.308132 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.308135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.308923 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.308933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.309550 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.309553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.309628 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.310029 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.310043 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.311325 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.313066 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.313070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.313072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.313339 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.313657 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.313660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.313662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.313744 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.314154 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.314157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.314235 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.314304 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.314417 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:44:38.315064 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.305836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.18.235.190:35741)
0927 18:44:38.305924 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.305927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.306143 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:44:38.306231 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.306234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.306275 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.306336 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.307226 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.307952 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.309285 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.309599 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.309685 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.311262 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.311266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.311268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.311292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.312986 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.313468 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.313471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.313473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.313551 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.313832 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.313835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.313837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.314024 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.314141 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.314807 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.314920 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:44:38.315140 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.306342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.18.235.188:36261)
0927 18:44:38.306628 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.306637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.306649 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:44:38.306680 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.306682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.306718 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.307188 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.307279 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.308009 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.309042 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.309410 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.309808 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.311456 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.311459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.311462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.311500 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.312957 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.313452 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.313455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.313457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.313556 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.313848 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.313850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.313852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.314049 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.314806 (+ 757us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.314874 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.314973 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260927 18:44:38.316715 426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.316969 426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:38.317040 426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460232dba1fb436f84ddf86e9f2cdfe5, State: Initialized, Role: FOLLOWER
I20260927 18:44:38.317061 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.305325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.18.235.189:37555)
0927 18:44:38.305500 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.305504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.306385 (+ 881us) server_negotiation.cc:407] Connection header received
0927 18:44:38.306529 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.306532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.306570 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.306630 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.307345 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.308038 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.309902 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.310191 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.310258 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.312519 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.312522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.312525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.312548 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.313938 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.314347 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.314350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.314352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.314433 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.315124 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.315128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.315130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.315321 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.315425 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.316820 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.316924 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 18:44:38.317523 426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.317963 351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: Bootstrap starting.
I20260927 18:44:38.317970 441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.318224 441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:38.318791 426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:38.319577 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.305781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.18.235.188:36261)
0927 18:44:38.305938 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.305942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.307117 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 18:44:38.307286 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.307289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.307323 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.307364 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.308152 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.308844 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.311070 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.311353 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.311421 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.315998 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.316001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.316003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.316025 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.317699 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.318373 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.318376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.318378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.318475 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.318786 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.318789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.318791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.319011 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.319133 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.319357 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.319450 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 18:44:38.319665 32744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } attempt: 3
I20260927 18:44:38.320592 351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:38.322275 351 log.cc:824] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:44:38 dist-test-slave-6lr6 krb5kdc[32663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534677, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
W20260927 18:44:38.324740 452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:38.324841 452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:38.322338 358 master_runner.cc:186] Error getting master registration for 127.18.235.190:35741: OK, instance_id { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" instance_seqno: 1790534677865076 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:44:38.325727 351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: No bootstrap required, opened a new log
I20260927 18:44:38.325805 358 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:44:38.327903 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.319970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:36261 (local address 127.0.0.1:48434)
0927 18:44:38.320061 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:38.320074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:38.320142 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:38.320439 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:38.320442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:38.320602 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:38.320685 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.320692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.321564 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.321567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.322293 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:38.322304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.322889 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.322892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:38.322968 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.323382 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:38.323396 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:38.324703 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:38.326448 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.326450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.326452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.326669 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.326951 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:38.326954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:38.326956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:38.327038 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:38.327448 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:38.327451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:38.327535 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 18:44:38.327632 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:44:38.327751 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:44:38.328418 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.320195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.18.235.188:36261)
0927 18:44:38.320274 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.320278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.320288 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:44:38.320315 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.320317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.320345 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.320403 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.320803 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.321448 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.322420 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.322705 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.322790 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.324843 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.324847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.324849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.324872 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.326335 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.326774 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.326778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.326779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.326856 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.327142 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.327145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.327146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.327323 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.327404 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.328190 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.328288 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:44:38.328490 351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.328687 351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:38.328740 351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd9df0db252450dbf181086e14fc3ea, State: Initialized, Role: FOLLOWER
I20260927 18:44:38.329909 351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.330309 455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.330466 455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:38.330992 351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:38.331099 32744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: Bootstrap starting.
I20260927 18:44:38.333530 32744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:38.334347 32744 log.cc:824] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: Log is configured to *not* fsync() on all Append() calls
W20260927 18:44:38.335937 466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:38.336052 466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:38.333660 32751 master_runner.cc:186] Error getting master registration for 127.18.235.190:35741: OK, instance_id { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" instance_seqno: 1790534677865076 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:44:38.337055 32751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:44:38.342480 32744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: No bootstrap required, opened a new log
I20260927 18:44:38.345167 32744 raft_consensus.cc:348] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.345336 32744 raft_consensus.cc:374] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:38.345383 32744 raft_consensus.cc:729] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70e3673be9b4d95b444c63f13a1aca4, State: Initialized, Role: FOLLOWER
I20260927 18:44:38.345743 32744 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.346868 32744 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:38.347548 469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.347723 469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:44:38.351344 480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:38.351465 480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:38.355674 32676 master_runner.cc:263] No leader master found from master a70e3673be9b4d95b444c63f13a1aca4
I20260927 18:44:38.355739 32676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 18:44:38.361939 32697 tablet.cc:2369] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:38.511083 379 tablet.cc:2369] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:38.514943 469 raft_consensus.cc:482] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:38.515118 469 raft_consensus.cc:504] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.516119 469 leader_election.cc:288] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bd9df0db252450dbf181086e14fc3ea (127.18.235.189:37555), 460232dba1fb436f84ddf86e9f2cdfe5 (127.18.235.188:36261)
I20260927 18:44:38.516551 400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a70e3673be9b4d95b444c63f13a1aca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" is_pre_election: true
I20260927 18:44:38.516556 325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a70e3673be9b4d95b444c63f13a1aca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd9df0db252450dbf181086e14fc3ea" is_pre_election: true
I20260927 18:44:38.516836 325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70e3673be9b4d95b444c63f13a1aca4 in term 0.
I20260927 18:44:38.516836 400 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70e3673be9b4d95b444c63f13a1aca4 in term 0.
I20260927 18:44:38.517237 32694 leader_election.cc:302] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 460232dba1fb436f84ddf86e9f2cdfe5, 8bd9df0db252450dbf181086e14fc3ea, a70e3673be9b4d95b444c63f13a1aca4; no voters:
I20260927 18:44:38.517493 469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:38.517575 469 raft_consensus.cc:482] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:38.517622 469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:38.518563 469 raft_consensus.cc:504] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.518846 469 leader_election.cc:288] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [CANDIDATE]: Term 1 election: Requested vote from peers 8bd9df0db252450dbf181086e14fc3ea (127.18.235.189:37555), 460232dba1fb436f84ddf86e9f2cdfe5 (127.18.235.188:36261)
I20260927 18:44:38.519124 400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a70e3673be9b4d95b444c63f13a1aca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460232dba1fb436f84ddf86e9f2cdfe5"
I20260927 18:44:38.519182 325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a70e3673be9b4d95b444c63f13a1aca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd9df0db252450dbf181086e14fc3ea"
I20260927 18:44:38.519268 400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:38.519281 325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:38.520377 400 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70e3673be9b4d95b444c63f13a1aca4 in term 1.
I20260927 18:44:38.520377 325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70e3673be9b4d95b444c63f13a1aca4 in term 1.
I20260927 18:44:38.520643 32694 leader_election.cc:302] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 460232dba1fb436f84ddf86e9f2cdfe5, a70e3673be9b4d95b444c63f13a1aca4; no voters:
I20260927 18:44:38.520835 469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:38.521041 469 raft_consensus.cc:686] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [term 1 LEADER]: Becoming Leader. State: Replica: a70e3673be9b4d95b444c63f13a1aca4, State: Running, Role: LEADER
I20260927 18:44:38.521366 469 consensus_queue.cc:237] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } }
I20260927 18:44:38.523036 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70e3673be9b4d95b444c63f13a1aca4. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.523147 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:38.523509 484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:38.524235 484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:38.528231 325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [term 1 FOLLOWER]: Refusing update from remote peer a70e3673be9b4d95b444c63f13a1aca4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:38.528385 400 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [term 1 FOLLOWER]: Refusing update from remote peer a70e3673be9b4d95b444c63f13a1aca4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:38.528786 482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:38.528919 469 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:38.532315 441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70e3673be9b4d95b444c63f13a1aca4. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.532425 441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:38.532424 455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70e3673be9b4d95b444c63f13a1aca4. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.532544 455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:38.536665 469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.536769 469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:38.536999 441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.537120 441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:38.537374 469 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.537472 469 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:38.537986 455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a70e3673be9b4d95b444c63f13a1aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a70e3673be9b4d95b444c63f13a1aca4" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35741 } } peers { permanent_uuid: "8bd9df0db252450dbf181086e14fc3ea" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 37555 } } peers { permanent_uuid: "460232dba1fb436f84ddf86e9f2cdfe5" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 36261 } } }
I20260927 18:44:38.538094 455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:38.539525 484 catalog_manager.cc:1347] Generated new cluster ID: 49812dd892ec433e85686ee8d527d778
I20260927 18:44:38.539582 484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:38.551409 484 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:38.552505 484 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:38.563922 484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a70e3673be9b4d95b444c63f13a1aca4: Generated new TSK 0
I20260927 18:44:38.564575 484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:39.326046 452 catalog_manager.cc:1259] Loaded cluster ID: 49812dd892ec433e85686ee8d527d778
I20260927 18:44:39.326121 452 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: loading cluster ID for follower catalog manager: success
I20260927 18:44:39.327713 452 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: acquiring CA information for follower catalog manager: success
I20260927 18:44:39.328811 452 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 460232dba1fb436f84ddf86e9f2cdfe5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:44:39.337044 466 catalog_manager.cc:1259] Loaded cluster ID: 49812dd892ec433e85686ee8d527d778
I20260927 18:44:39.337122 466 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: loading cluster ID for follower catalog manager: success
I20260927 18:44:39.338573 466 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: acquiring CA information for follower catalog manager: success
I20260927 18:44:39.339538 466 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bd9df0db252450dbf181086e14fc3ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:44:39.620085 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 32676
I20260927 18:44:39.625535 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 32751
I20260927 18:44:39.630098 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 358
2026-09-27T18:44:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2982 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:39 dist-test-slave-6lr6 krb5kdc[500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:39 dist-test-slave-6lr6 krb5kdc[500](info): set up 2 sockets
Sep 27 18:44:39 dist-test-slave-6lr6 krb5kdc[500](info): commencing operation
krb5kdc: starting...
W20260927 18:44:40.580921 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:44:40 dist-test-slave-6lr6 krb5kdc[500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:44:40.685096 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45389
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:40.788928 513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:40.789251 513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:40.789292 513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:40.793151 513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:40.793197 513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:40.793215 513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:40.793228 513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:40.793258 513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:40.798235 513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:45389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:40.799129 513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:40.800060 513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:40.807834 519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:40.807824 521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:40.807824 518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:40.808182 513 server_base.cc:1034] running on GCE node
I20260927 18:44:40.808691 513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:40.809751 513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:40.810906 513 hybrid_clock.cc:648] HybridClock initialized: now 1790534680810887 us; error 31 us; skew 500 ppm
Sep 27 18:44:40 dist-test-slave-6lr6 krb5kdc[500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:40.817873 513 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:40.818694 513 webserver.cc:460] Webserver started at http://127.18.235.190:35175/ using document root <none> and password file <none>
I20260927 18:44:40.819186 513 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:40.819250 513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:40.819434 513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:40.821380 513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "a89c8a85489643f8aee8ad7fc0c2872e"
format_stamp: "Formatted at 2026-09-27 18:44:40 on dist-test-slave-6lr6"
server_key: "76c745c6a1e709291c646849793bc914"
server_key_iv: "f3db574c4e6fe543ffc049a865eae25b"
server_key_version: "encryptionkey@0"
I20260927 18:44:40.821877 513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "a89c8a85489643f8aee8ad7fc0c2872e"
format_stamp: "Formatted at 2026-09-27 18:44:40 on dist-test-slave-6lr6"
server_key: "76c745c6a1e709291c646849793bc914"
server_key_iv: "f3db574c4e6fe543ffc049a865eae25b"
server_key_version: "encryptionkey@0"
I20260927 18:44:40.825266 513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:40.827361 528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:40.828406 513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:40.828522 513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "a89c8a85489643f8aee8ad7fc0c2872e"
format_stamp: "Formatted at 2026-09-27 18:44:40 on dist-test-slave-6lr6"
server_key: "76c745c6a1e709291c646849793bc914"
server_key_iv: "f3db574c4e6fe543ffc049a865eae25b"
server_key_version: "encryptionkey@0"
I20260927 18:44:40.828606 513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:40.837756 513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:40.840072 513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:40.840216 513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:40.846227 513 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:45389
I20260927 18:44:40.846240 580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:45389 every 8 connection(s)
I20260927 18:44:40.847010 513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:40.849396 581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.849871 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 513
I20260927 18:44:40.849963 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:44:40.850198 19374 external_mini_cluster.cc:1428] Setting key 5ced6fec8bcd2303364e42635311e33e
I20260927 18:44:40.855690 581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e: Bootstrap starting.
I20260927 18:44:40.857944 581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e: Neither blocks nor log segments found. Creating new log.
Sep 27 18:44:40 dist-test-slave-6lr6 krb5kdc[500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:40.858606 581 log.cc:824] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:40.860524 581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e: No bootstrap required, opened a new log
I20260927 18:44:40.863590 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.851294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.18.235.190:45389)
0927 18:44:40.851678 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.851695 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.851735 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:44:40.852410 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.852432 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.852826 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.853182 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:40.853781 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.855174 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.856015 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.856662 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.856811 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.858925 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:40.858929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:40.858940 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:40.858963 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:40.860704 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.861268 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.861274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.861279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.861412 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.861645 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.861648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.861649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.862090 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:40.862269 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.862459 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.862753 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260927 18:44:40.864612 581 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45389 } }
I20260927 18:44:40.864814 581 raft_consensus.cc:374] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.864859 581 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89c8a85489643f8aee8ad7fc0c2872e, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.865293 581 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45389 } }
I20260927 18:44:40.865443 581 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:40.865509 581 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:40.865597 581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:40.866137 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:40.866442 581 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45389 } }
I20260927 18:44:40.866729 581 leader_election.cc:302] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89c8a85489643f8aee8ad7fc0c2872e; no voters:
I20260927 18:44:40.867002 581 leader_election.cc:288] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:40.867060 586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:40.867241 586 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [term 1 LEADER]: Becoming Leader. State: Replica: a89c8a85489643f8aee8ad7fc0c2872e, State: Running, Role: LEADER
I20260927 18:44:40.867549 586 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45389 } }
I20260927 18:44:40.868117 581 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:40.869187 588 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89c8a85489643f8aee8ad7fc0c2872e. Latest consensus state: current_term: 1 leader_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45389 } } }
I20260927 18:44:40.869201 587 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89c8a85489643f8aee8ad7fc0c2872e" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45389 } } }
I20260927 18:44:40.869385 588 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:40.869472 587 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:40.870260 597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:40.872880 597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:40.878108 597 catalog_manager.cc:1347] Generated new cluster ID: 32d29f7157374af1a95124fcb2cef8c5
I20260927 18:44:40.878155 597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:40.901906 597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:40.902667 597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:40.906584 597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a89c8a85489643f8aee8ad7fc0c2872e: Generated new TSK 0
I20260927 18:44:40.907099 597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:40.920920 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 513
2026-09-27T18:44:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1289 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:40 dist-test-slave-6lr6 krb5kdc[608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:40 dist-test-slave-6lr6 krb5kdc[608](info): set up 2 sockets
Sep 27 18:44:40 dist-test-slave-6lr6 krb5kdc[608](info): commencing operation
krb5kdc: starting...
W20260927 18:44:41.856880 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:44:41 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:44:41.959503 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32841
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:42.062306 621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:42.062541 621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:42.062587 621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:42.066463 621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:42.066506 621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:42.066542 621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:42.066570 621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:42.066597 621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:42.071451 621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:42.072288 621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:42.073167 621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:42.080475 626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:42.080590 629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:42.080566 621 server_base.cc:1034] running on GCE node
W20260927 18:44:42.080468 627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:42.081249 621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:42.082151 621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:42.083295 621 hybrid_clock.cc:648] HybridClock initialized: now 1790534682083276 us; error 33 us; skew 500 ppm
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:42.085736 621 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:42.086496 621 webserver.cc:460] Webserver started at http://127.18.235.190:37173/ using document root <none> and password file <none>
I20260927 18:44:42.086961 621 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:42.087014 621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:42.087188 621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:42.088950 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "0f87b9ab28c34c619ba5ab5feebfa920"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "b3dfc4c77065ba1d8815676bc48a8fd4"
server_key_iv: "f5d7ac91abd9419483090dcd66c9b5dc"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.089584 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "0f87b9ab28c34c619ba5ab5feebfa920"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "b3dfc4c77065ba1d8815676bc48a8fd4"
server_key_iv: "f5d7ac91abd9419483090dcd66c9b5dc"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.092770 621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:42.094897 636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.095959 621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:42.096072 621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "0f87b9ab28c34c619ba5ab5feebfa920"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "b3dfc4c77065ba1d8815676bc48a8fd4"
server_key_iv: "f5d7ac91abd9419483090dcd66c9b5dc"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.096159 621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:42.106964 621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:42.109580 621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:42.109720 621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:42.115710 621 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:32841
I20260927 18:44:42.115948 688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:32841 every 8 connection(s)
I20260927 18:44:42.116518 621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:42.118916 689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:42.123860 689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap starting.
I20260927 18:44:42.124035 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 621
I20260927 18:44:42.124155 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:44:42.124429 19374 external_mini_cluster.cc:1428] Setting key 99f5eeed5a4f9037a23f4d41eea0a5fe
I20260927 18:44:42.126264 689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:42.126911 689 log.cc:824] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:42.128738 689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: No bootstrap required, opened a new log
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:42.132697 689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:42.132916 689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:42.132972 689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f87b9ab28c34c619ba5ab5feebfa920, State: Initialized, Role: FOLLOWER
I20260927 18:44:42.133383 689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:42.133508 689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:42.133569 689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:42.133669 689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:42.134403 689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:42.134699 689 leader_election.cc:302] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f87b9ab28c34c619ba5ab5feebfa920; no voters:
I20260927 18:44:42.134949 689 leader_election.cc:288] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:42.135018 694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:42.135174 694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 LEADER]: Becoming Leader. State: Replica: 0f87b9ab28c34c619ba5ab5feebfa920, State: Running, Role: LEADER
I20260927 18:44:42.135484 694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:42.136092 689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:42.136979 695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } } }
I20260927 18:44:42.137096 695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:42.137463 696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f87b9ab28c34c619ba5ab5feebfa920. Latest consensus state: current_term: 1 leader_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } } }
I20260927 18:44:42.137601 696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:42.137795 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.125631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.18.235.190:32841)
0927 18:44:42.126038 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.126051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.126083 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:44:42.126681 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.126698 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.127001 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.127298 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:42.128037 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.128993 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.130140 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.130567 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.130673 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.133213 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:42.133218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:42.133233 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:42.133263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:42.134744 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.135166 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.135172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.135178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.135310 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.135607 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.135610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.135612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.135966 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:42.136154 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.136413 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.136811 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260927 18:44:42.138209 703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:42.140599 703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:42.145437 703 catalog_manager.cc:1347] Generated new cluster ID: d0f6da3ebe4d4600b9e7fd51e2e5df0f
I20260927 18:44:42.145493 703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:42.151721 703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:42.152786 703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:42.156575 703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Generated new TSK 0
I20260927 18:44:42.157078 703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:44:42.180060 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32841
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:42.282351 715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:42.282581 715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:42.282626 715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:42.286607 715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:42.286657 715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:42.286729 715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:44:42.291693 715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:42.292588 715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:42.293534 715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:44:42.302390 715 server_base.cc:1034] running on GCE node
W20260927 18:44:42.302359 720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:42.302366 723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:42.302358 721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:42.302917 715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:42.303442 715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:42.304589 715 hybrid_clock.cc:648] HybridClock initialized: now 1790534682304571 us; error 30 us; skew 500 ppm
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:42.307044 715 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:44:42.307816 715 webserver.cc:460] Webserver started at http://127.18.235.129:36391/ using document root <none> and password file <none>
I20260927 18:44:42.308285 715 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:42.308341 715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:42.308533 715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:42.310365 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "c339e88fcb9a498e8bbea3a633e16253"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "2bdf6add00f261e180add87b2309586e"
server_key_iv: "a688be90a9e49d71ee618fbd32ffec95"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.310856 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "c339e88fcb9a498e8bbea3a633e16253"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "2bdf6add00f261e180add87b2309586e"
server_key_iv: "a688be90a9e49d71ee618fbd32ffec95"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.314083 715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 18:44:42.316219 730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.317317 715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:42.317412 715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "c339e88fcb9a498e8bbea3a633e16253"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "2bdf6add00f261e180add87b2309586e"
server_key_iv: "a688be90a9e49d71ee618fbd32ffec95"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.317495 715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:42.325526 715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:42.328148 715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:42.328293 715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:42.328769 715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:42.329469 715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:42.329512 715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.329566 715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:42.329594 715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.337370 715 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:40299
I20260927 18:44:42.337458 843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:40299 every 8 connection(s)
I20260927 18:44:42.338140 715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:42.345332 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 715
I20260927 18:44:42.345431 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:42.345681 19374 external_mini_cluster.cc:1428] Setting key 01f540f72ad84bcbaa87f25109237244
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:42.349885 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.339630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:59575 (local address 127.18.235.190:32841)
0927 18:44:42.339761 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.339765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.340499 (+ 734us) server_negotiation.cc:407] Connection header received
0927 18:44:42.341730 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.341733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.341778 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.341844 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:42.342830 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.343344 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.344339 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.344555 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.344619 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.346834 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:42.346838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:42.346840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:42.346867 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:42.348037 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.348565 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.348568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.348569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.348647 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.348952 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.348955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.348956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.349133 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:42.349207 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.349682 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.349774 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:44:42.350817 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.339973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:59575)
0927 18:44:42.340364 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.340401 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.341550 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.341977 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.341983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.342286 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:42.342720 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.342733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.343426 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.343428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.344255 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.344263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.344659 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.344661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.344787 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.345250 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:42.345265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:42.346686 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:42.348138 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:42.348143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:42.348156 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:42.348465 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:42.348725 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:42.348728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:42.348729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:42.348881 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:42.349212 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:42.349218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:42.349481 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.349666 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.349941 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260927 18:44:42.351922 844 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32841
I20260927 18:44:42.352109 844 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:42.352506 844 heartbeater.cc:511] Master 127.18.235.190:32841 requested a full tablet report, sending...
I20260927 18:44:42.353881 653 ts_manager.cc:194] Registered new tserver with Master: c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299)
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:44:42.354938 653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:59575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:44:42.368707 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32841
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:42.472324 849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:42.472537 849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:42.472592 849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:42.476418 849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:42.476465 849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:42.476541 849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:44:42.481388 849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:42.482224 849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:42.483095 849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:42.491518 857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:42.491549 854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:42.491649 849 server_base.cc:1034] running on GCE node
W20260927 18:44:42.491531 855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:42.492194 849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:42.492738 849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:42.493871 849 hybrid_clock.cc:648] HybridClock initialized: now 1790534682493856 us; error 26 us; skew 500 ppm
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:42.496260 849 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:44:42.496984 849 webserver.cc:460] Webserver started at http://127.18.235.130:34093/ using document root <none> and password file <none>
I20260927 18:44:42.497467 849 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:42.497532 849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:42.497712 849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:42.499545 849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "aeb505d6786f47108374a2782964e9a5"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "a62a9904b538b0a5b920d466496f473e"
server_key_iv: "58cfd41d450f05be3b974348c5732bfe"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.500048 849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "aeb505d6786f47108374a2782964e9a5"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "a62a9904b538b0a5b920d466496f473e"
server_key_iv: "58cfd41d450f05be3b974348c5732bfe"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.503302 849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:44:42.505307 864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.506343 849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:42.506444 849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "aeb505d6786f47108374a2782964e9a5"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "a62a9904b538b0a5b920d466496f473e"
server_key_iv: "58cfd41d450f05be3b974348c5732bfe"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.506533 849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:42.529054 849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:42.531653 849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:42.531797 849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:42.532258 849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:42.532949 849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:42.532999 849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.533046 849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:42.533075 849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.540902 849 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42179
I20260927 18:44:42.540961 977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42179 every 8 connection(s)
I20260927 18:44:42.541638 849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:42.543218 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 849
I20260927 18:44:42.543308 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:42.543540 19374 external_mini_cluster.cc:1428] Setting key 8c00b32e9f129a8f930afe4c63456d14
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:44:42.554744 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.544401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:58405 (local address 127.18.235.190:32841)
0927 18:44:42.544540 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.544544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.544560 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:42.545563 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.545567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.545623 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.545702 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:42.546755 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.547281 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.548793 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.549058 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.549181 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.551595 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:42.551598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:42.551600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:42.551623 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:42.552883 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.553383 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.553385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.553387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.553461 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.553755 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.553759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.553760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.553945 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:42.554042 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.554474 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.554590 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:44:42.555440 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.543521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:58405)
0927 18:44:42.544000 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.544050 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.545361 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.545849 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.545855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.546180 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:42.546603 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.546616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.547415 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.547418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.548671 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.548684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.549225 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.549230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.549434 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.549941 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:42.549964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:42.551419 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:42.552984 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:42.552990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:42.553002 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:42.553292 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:42.553540 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:42.553543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:42.553545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:42.553674 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:42.554050 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:42.554056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:42.554264 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.554412 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.554696 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260927 18:44:42.556543 978 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32841
I20260927 18:44:42.556752 978 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:42.557241 978 heartbeater.cc:511] Master 127.18.235.190:32841 requested a full tablet report, sending...
I20260927 18:44:42.558221 653 ts_manager.cc:194] Registered new tserver with Master: aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
I20260927 18:44:42.558806 653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:58405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:44:42.569068 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32841
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:42.669857 983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:42.670059 983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:42.670112 983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:42.673933 983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:42.673978 983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:42.674053 983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:44:42.678918 983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:42.679742 983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:42.680598 983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:42.688898 989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:42.688905 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:42.688951 988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:42.689249 983 server_base.cc:1034] running on GCE node
I20260927 18:44:42.689586 983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:42.690135 983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:42.691282 983 hybrid_clock.cc:648] HybridClock initialized: now 1790534682691264 us; error 27 us; skew 500 ppm
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:42.693638 983 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:44:42.694348 983 webserver.cc:460] Webserver started at http://127.18.235.131:35341/ using document root <none> and password file <none>
I20260927 18:44:42.694803 983 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:42.694857 983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:42.695006 983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:42.696774 983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "1fef89d91c6a44048143ef9c4c02e873"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "3c8a97697d37faedba82c943d51929f7"
server_key_iv: "0ee3d3497fa988ef6917be14313270fb"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.697296 983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "1fef89d91c6a44048143ef9c4c02e873"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "3c8a97697d37faedba82c943d51929f7"
server_key_iv: "0ee3d3497fa988ef6917be14313270fb"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.700654 983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:44:42.702735 998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.703773 983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:44:42.703871 983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "1fef89d91c6a44048143ef9c4c02e873"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "3c8a97697d37faedba82c943d51929f7"
server_key_iv: "0ee3d3497fa988ef6917be14313270fb"
server_key_version: "encryptionkey@0"
I20260927 18:44:42.703951 983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:42.716727 983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:42.719417 983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:42.719568 983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:42.720046 983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:42.720784 983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:42.720829 983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.720875 983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:42.720901 983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:42.728555 983 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:43199
I20260927 18:44:42.728608 1111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:43199 every 8 connection(s)
I20260927 18:44:42.729336 983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:42.733325 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 983
I20260927 18:44:42.733415 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:42.733639 19374 external_mini_cluster.cc:1428] Setting key 16a0bd43571dd0c790a8e369ff3303dd
Sep 27 18:44:42 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534682, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:42.740907 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.730838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:50673 (local address 127.18.235.190:32841)
0927 18:44:42.730950 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.730954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.731662 (+ 708us) server_negotiation.cc:407] Connection header received
0927 18:44:42.732874 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.732877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.732921 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.732988 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:42.734074 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.734760 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.735917 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.736175 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.736282 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.738384 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:42.738387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:42.738389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:42.738413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:42.739462 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.739847 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.739849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.739851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.739913 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.740148 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.740151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.740152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.740298 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:42.740364 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.740716 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.740800 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:44:42.741614 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.731155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:50673)
0927 18:44:42.731532 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.731567 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.732708 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.733148 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.733153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.733485 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:42.733933 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.733946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.734869 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.734872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.735816 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.735825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.736319 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.736322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.736451 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.736859 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:42.736874 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:42.738233 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:42.739547 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:42.739552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:42.739561 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:42.739778 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:42.739970 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:42.739972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:42.739973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:42.740087 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:42.740362 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:42.740366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:42.740551 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.740685 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.740888 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260927 18:44:42.742509 1112 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32841
I20260927 18:44:42.742697 1112 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:42.743085 1112 heartbeater.cc:511] Master 127.18.235.190:32841 requested a full tablet report, sending...
I20260927 18:44:42.743921 653 ts_manager.cc:194] Registered new tserver with Master: 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199)
I20260927 18:44:42.744434 653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:50673
I20260927 18:44:42.746856 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:44:42.754127 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.748459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.18.235.190:32841)
0927 18:44:42.748574 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.748578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.748704 (+ 126us) server_negotiation.cc:407] Connection header received
0927 18:44:42.748786 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.748788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.748821 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.748878 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:42.749471 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.749935 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.750658 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.750844 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.750906 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.751942 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:42.751945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:42.751946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:42.751960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:42.752944 (+ 984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.753296 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.753298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.753299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.753360 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:42.753543 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:42.753545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:42.753546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:42.753715 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:42.753804 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.753934 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.754007 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:44:42.756323 653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:42.759105 653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:42.771363 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.766187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40299 (local address 127.0.0.1:41816)
0927 18:44:42.766495 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.766539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.766659 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.767456 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.767470 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.767493 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:42.767602 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.767608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.768805 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.768808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.769904 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.769915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.770885 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.770888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.770998 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.771032 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.771128 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.771238 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260927 18:44:42.771382 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.766325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.18.235.129:40299)
0927 18:44:42.766969 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.766981 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.767011 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:44:42.767088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.767094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.767286 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.767747 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:42.767904 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.768689 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.770038 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.770780 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.770885 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.771011 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.771164 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.771250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":92,"threads_started":1}
I20260927 18:44:42.771883 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.766919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:43199 (local address 127.0.0.1:44416)
0927 18:44:42.767134 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.767155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.767234 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.768118 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.768121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.768140 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:42.768222 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.768228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.769595 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.769598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.770524 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.770535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.771530 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.771533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.771636 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.771658 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.771701 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.771764 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260927 18:44:42.772058 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.766352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42179 (local address 127.0.0.1:50982)
0927 18:44:42.766790 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.766803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.766883 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.767906 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.767909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.767926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:42.768009 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.768015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.769394 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.769397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.770474 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.770484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.771677 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.771681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.771772 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.771794 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.771868 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.771931 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":97,"threads_started":1}
I20260927 18:44:42.772116 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.766925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.18.235.130:42179)
0927 18:44:42.767464 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.767474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.767495 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:44:42.767564 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.767569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.767749 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.767925 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:42.768105 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.769264 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.770590 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.771399 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.771685 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.771806 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.771889 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.771977 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":119,"threads_started":1}
I20260927 18:44:42.774309 778 tablet_service.cc:1467] Processing CreateTablet for tablet 1e281711175443f1b5fef2c66fdfff76 (DEFAULT_TABLE table=test-table [id=f61783c9f56c4551a501dd06dfe69b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:42.774905 912 tablet_service.cc:1467] Processing CreateTablet for tablet 1e281711175443f1b5fef2c66fdfff76 (DEFAULT_TABLE table=test-table [id=f61783c9f56c4551a501dd06dfe69b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:42.775383 778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e281711175443f1b5fef2c66fdfff76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:42.775879 912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e281711175443f1b5fef2c66fdfff76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:42.781363 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.767271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.18.235.131:43199)
0927 18:44:42.767728 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.767738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.767756 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:44:42.767815 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.767820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.767978 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.768362 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:42.768476 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.769488 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.770647 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.771399 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.772141 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.772267 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.772310 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.781231 (+ 8921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":131,"threads_started":1}
I20260927 18:44:42.782403 1129 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Bootstrap starting.
I20260927 18:44:42.783404 1130 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Bootstrap starting.
I20260927 18:44:42.784056 1046 tablet_service.cc:1467] Processing CreateTablet for tablet 1e281711175443f1b5fef2c66fdfff76 (DEFAULT_TABLE table=test-table [id=f61783c9f56c4551a501dd06dfe69b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:42.784997 1046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e281711175443f1b5fef2c66fdfff76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:42.785264 1129 tablet_bootstrap.cc:654] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:42.785353 1130 tablet_bootstrap.cc:654] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:42.785969 1130 log.cc:824] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:42.786042 1129 log.cc:824] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:42.787781 1129 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: No bootstrap required, opened a new log
I20260927 18:44:42.787806 1130 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: No bootstrap required, opened a new log
I20260927 18:44:42.787951 1129 ts_tablet_manager.cc:1397] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:44:42.787971 1130 ts_tablet_manager.cc:1397] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:44:42.790604 1129 raft_consensus.cc:348] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.790793 1129 raft_consensus.cc:374] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:42.790850 1129 raft_consensus.cc:729] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb505d6786f47108374a2782964e9a5, State: Initialized, Role: FOLLOWER
I20260927 18:44:42.791234 1129 consensus_queue.cc:260] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.791599 1133 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Bootstrap starting.
I20260927 18:44:42.791519 1130 raft_consensus.cc:348] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.791754 1130 raft_consensus.cc:374] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:42.791814 1130 raft_consensus.cc:729] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c339e88fcb9a498e8bbea3a633e16253, State: Initialized, Role: FOLLOWER
I20260927 18:44:42.792155 1129 ts_tablet_manager.cc:1428] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 18:44:42.792239 1130 consensus_queue.cc:260] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.792438 978 heartbeater.cc:503] Master 127.18.235.190:32841 was elected leader, sending a full tablet report...
W20260927 18:44:42.792460 979 tablet.cc:2369] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:42.793154 1130 ts_tablet_manager.cc:1428] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 18:44:42.793200 844 heartbeater.cc:503] Master 127.18.235.190:32841 was elected leader, sending a full tablet report...
I20260927 18:44:42.794021 1133 tablet_bootstrap.cc:654] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:42.795018 1133 log.cc:824] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:42.796715 1133 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: No bootstrap required, opened a new log
I20260927 18:44:42.796866 1133 ts_tablet_manager.cc:1397] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:42.799463 1133 raft_consensus.cc:348] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.799645 1133 raft_consensus.cc:374] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:42.799692 1133 raft_consensus.cc:729] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fef89d91c6a44048143ef9c4c02e873, State: Initialized, Role: FOLLOWER
I20260927 18:44:42.799975 1133 consensus_queue.cc:260] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.800491 1112 heartbeater.cc:503] Master 127.18.235.190:32841 was elected leader, sending a full tablet report...
I20260927 18:44:42.800634 1133 ts_tablet_manager.cc:1428] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 18:44:42.839062 845 tablet.cc:2369] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:42.886417 1137 raft_consensus.cc:482] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:42.886596 1137 raft_consensus.cc:504] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.887604 1137 leader_election.cc:288] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299), aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
I20260927 18:44:42.891078 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.887799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40299 (local address 127.18.235.131:39705)
0927 18:44:42.887943 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.887957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.888039 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.888291 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.888294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.888311 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:42.888423 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.888429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.889262 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.889266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.890062 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.890071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.890704 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.890708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.890804 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.890825 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.890886 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.890949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 18:44:42.891113 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.888012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:33775 (local address 127.18.235.130:42179)
0927 18:44:42.888126 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.888130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.888274 (+ 144us) server_negotiation.cc:407] Connection header received
0927 18:44:42.888403 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.888406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.888452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.888517 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:42.888756 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.889291 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.890260 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.890636 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.890749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.890795 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.890959 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.891019 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:44:42.891134 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.887828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42179 (local address 127.18.235.131:33775)
0927 18:44:42.888202 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.888214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:42.888299 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:42.888537 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:42.888540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:42.888556 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:42.888658 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.888664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.889386 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.889388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.890148 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:42.890158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.890757 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.890761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:42.890849 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.890866 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:44:42.890938 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:44:42.890999 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":112,"threads_started":1}
I20260927 18:44:42.891124 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.887887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:39705 (local address 127.18.235.129:40299)
0927 18:44:42.888004 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.888008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.888024 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:42.888142 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.888145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.888197 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.888281 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:42.888532 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.889151 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.890192 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.890601 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.890690 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.890739 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.890918 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.890994 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:44:42.891758 798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e281711175443f1b5fef2c66fdfff76" candidate_uuid: "1fef89d91c6a44048143ef9c4c02e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c339e88fcb9a498e8bbea3a633e16253" is_pre_election: true
I20260927 18:44:42.891758 932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e281711175443f1b5fef2c66fdfff76" candidate_uuid: "1fef89d91c6a44048143ef9c4c02e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb505d6786f47108374a2782964e9a5" is_pre_election: true
I20260927 18:44:42.892045 932 raft_consensus.cc:2454] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fef89d91c6a44048143ef9c4c02e873 in term 0.
I20260927 18:44:42.892045 798 raft_consensus.cc:2454] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fef89d91c6a44048143ef9c4c02e873 in term 0.
I20260927 18:44:42.892504 1001 leader_election.cc:302] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fef89d91c6a44048143ef9c4c02e873, aeb505d6786f47108374a2782964e9a5, c339e88fcb9a498e8bbea3a633e16253; no voters:
I20260927 18:44:42.892719 1137 raft_consensus.cc:2781] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:42.892787 1137 raft_consensus.cc:482] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:42.892833 1137 raft_consensus.cc:3037] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:42.893787 1137 raft_consensus.cc:504] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.894074 1137 leader_election.cc:288] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [CANDIDATE]: Term 1 election: Requested vote from peers c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299), aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
I20260927 18:44:42.894347 798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e281711175443f1b5fef2c66fdfff76" candidate_uuid: "1fef89d91c6a44048143ef9c4c02e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c339e88fcb9a498e8bbea3a633e16253"
I20260927 18:44:42.894379 932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e281711175443f1b5fef2c66fdfff76" candidate_uuid: "1fef89d91c6a44048143ef9c4c02e873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb505d6786f47108374a2782964e9a5"
I20260927 18:44:42.894480 932 raft_consensus.cc:3037] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:42.894452 798 raft_consensus.cc:3037] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:42.895390 798 raft_consensus.cc:2454] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fef89d91c6a44048143ef9c4c02e873 in term 1.
I20260927 18:44:42.895411 932 raft_consensus.cc:2454] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fef89d91c6a44048143ef9c4c02e873 in term 1.
I20260927 18:44:42.895615 1001 leader_election.cc:302] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fef89d91c6a44048143ef9c4c02e873, c339e88fcb9a498e8bbea3a633e16253; no voters:
I20260927 18:44:42.895779 1137 raft_consensus.cc:2781] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:42.895964 1137 raft_consensus.cc:686] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 1 LEADER]: Becoming Leader. State: Replica: 1fef89d91c6a44048143ef9c4c02e873, State: Running, Role: LEADER
I20260927 18:44:42.896262 1137 consensus_queue.cc:237] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:42.899042 653 catalog_manager.cc:5495] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131). New cstate: current_term: 1 leader_uuid: "1fef89d91c6a44048143ef9c4c02e873" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } health_report { overall_health: HEALTHY } } }
I20260927 18:44:42.918349 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.915499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.18.235.131:43199)
0927 18:44:42.915601 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:42.915604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:42.915631 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:44:42.915769 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:42.915771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:42.915813 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:42.915888 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:42.916045 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.916491 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.917271 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:42.917462 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:42.917529 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:42.917716 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:42.918071 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:42.918150 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:42.918176 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 18:44:42.918215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:44:42.923300 932 raft_consensus.cc:1261] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Refusing update from remote peer 1fef89d91c6a44048143ef9c4c02e873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:42.923301 798 raft_consensus.cc:1261] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Refusing update from remote peer 1fef89d91c6a44048143ef9c4c02e873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:42.923923 1139 consensus_queue.cc:1035] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:42.924095 1137 consensus_queue.cc:1035] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:42.938987 652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:42.939110 652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:42.941200 652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Sending DeleteTablet for 3 replicas of tablet 1e281711175443f1b5fef2c66fdfff76
I20260927 18:44:42.941825 1046 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e281711175443f1b5fef2c66fdfff76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:42 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:44416
I20260927 18:44:42.941828 912 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e281711175443f1b5fef2c66fdfff76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:42 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:50982
I20260927 18:44:42.942265 1152 tablet_replica.cc:330] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: stopping tablet replica
I20260927 18:44:42.942255 778 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e281711175443f1b5fef2c66fdfff76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:42 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:41816
I20260927 18:44:42.942551 1152 raft_consensus.cc:2229] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:42.942687 1154 tablet_replica.cc:330] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: stopping tablet replica
I20260927 18:44:42.942878 1152 raft_consensus.cc:2258] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:42.942914 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 715
I20260927 18:44:42.943048 1154 raft_consensus.cc:2229] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:42.944197 1152 ts_tablet_manager.cc:1905] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:42.946791 1152 ts_tablet_manager.cc:1918] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:42.946883 1152 log.cc:1196] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/1e281711175443f1b5fef2c66fdfff76
I20260927 18:44:42.947191 1152 ts_tablet_manager.cc:1939] T 1e281711175443f1b5fef2c66fdfff76 P 1fef89d91c6a44048143ef9c4c02e873: Deleting consensus metadata
I20260927 18:44:42.947180 1153 tablet_replica.cc:330] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: stopping tablet replica
I20260927 18:44:42.947505 1153 raft_consensus.cc:2229] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:42.947820 1153 raft_consensus.cc:2258] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:42.948258 640 catalog_manager.cc:4841] TS 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199): tablet 1e281711175443f1b5fef2c66fdfff76 (table test-table [id=f61783c9f56c4551a501dd06dfe69b8b]) successfully deleted
I20260927 18:44:42.949180 1153 ts_tablet_manager.cc:1905] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:44:42.949697 639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:40299 (error 108)
I20260927 18:44:42.950562 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.950296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40299 (local address 127.0.0.1:41824)
0927 18:44:42.950409 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.950471 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:40299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:44:42.950620 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 849
W20260927 18:44:42.952808 639 catalog_manager.cc:4584] TS c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e281711175443f1b5fef2c66fdfff76: Network error: Client connection negotiation failed: client connection to 127.18.235.129:40299: connect: Connection refused (error 111)
I20260927 18:44:42.956208 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 983
I20260927 18:44:42.956355 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:42.956136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42179 (local address 127.0.0.1:50988)
0927 18:44:42.956247 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:42.956297 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.130:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:44:42.960656 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 621
I20260927 18:44:42.965566 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32841
--webserver_interface=127.18.235.190
--webserver_port=37173
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:43.068735 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:43.068953 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:43.069002 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:43.072808 1156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:43.072857 1156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:43.072882 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:43.072903 1156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:43.072927 1156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:43.077848 1156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=37173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:43.078684 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:43.079545 1156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:43.090838 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.090915 1156 server_base.cc:1034] running on GCE node
W20260927 18:44:43.090904 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:43.090824 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.091591 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:43.092597 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:43.093741 1156 hybrid_clock.cc:648] HybridClock initialized: now 1790534683093724 us; error 30 us; skew 500 ppm
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:43.096168 1156 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:43.096937 1156 webserver.cc:460] Webserver started at http://127.18.235.190:37173/ using document root <none> and password file <none>
I20260927 18:44:43.097455 1156 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:43.097537 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:43.100939 1156 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:43.102828 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.103922 1156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:43.104027 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "0f87b9ab28c34c619ba5ab5feebfa920"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "b3dfc4c77065ba1d8815676bc48a8fd4"
server_key_iv: "f5d7ac91abd9419483090dcd66c9b5dc"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.104575 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:43.115661 1156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:43.118525 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:43.118690 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:43.125579 1156 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:32841
I20260927 18:44:43.125826 1224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:32841 every 8 connection(s)
I20260927 18:44:43.126577 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:43.130221 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1156
I20260927 18:44:43.130424 1225 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:44:43.130677 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:40299
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:32841
--webserver_port=36391
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:43.132063 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap starting.
I20260927 18:44:43.152271 1225 log.cc:824] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:43.159993 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:44:43.160269 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap complete.
I20260927 18:44:43.163838 1225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:43.164230 1225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f87b9ab28c34c619ba5ab5feebfa920, State: Initialized, Role: FOLLOWER
I20260927 18:44:43.164646 1225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:43.164769 1225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:43.164839 1225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:43.164927 1225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:44:43.166608 1225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:43.166900 1225 leader_election.cc:302] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f87b9ab28c34c619ba5ab5feebfa920; no voters:
I20260927 18:44:43.167148 1225 leader_election.cc:288] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:44:43.167251 1229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:44:43.167454 1229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 LEADER]: Becoming Leader. State: Replica: 0f87b9ab28c34c619ba5ab5feebfa920, State: Running, Role: LEADER
I20260927 18:44:43.167721 1229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:43.168239 1225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:43.168547 1229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f87b9ab28c34c619ba5ab5feebfa920. Latest consensus state: current_term: 2 leader_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } } }
I20260927 18:44:43.168655 1229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:43.169240 1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } } }
I20260927 18:44:43.169335 1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:43.169824 1233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:43.174189 1233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:43.174746 1233 catalog_manager.cc:1259] Loaded cluster ID: d0f6da3ebe4d4600b9e7fd51e2e5df0f
I20260927 18:44:43.174803 1233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:43.176091 1233 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 18:44:43.180703 1233 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 18:44:43.185428 1233 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:43.186079 1233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Loaded TSK: 0
I20260927 18:44:43.187204 1233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:44:43.244473 1227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:43.244678 1227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:43.244715 1227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:43.248594 1227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:43.248639 1227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:43.248697 1227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:44:43.253566 1227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:40299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=36391
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:43.254398 1227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:43.255259 1227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:43.264881 1253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:43.264886 1255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.265208 1227 server_base.cc:1034] running on GCE node
W20260927 18:44:43.265218 1252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.265694 1227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:43.266418 1227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:43.267612 1227 hybrid_clock.cc:648] HybridClock initialized: now 1790534683267590 us; error 29 us; skew 500 ppm
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:43.270627 1227 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:44:43.271405 1227 webserver.cc:460] Webserver started at http://127.18.235.129:36391/ using document root <none> and password file <none>
I20260927 18:44:43.271883 1227 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:43.271950 1227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:43.275534 1227 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:44:43.277637 1262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.278822 1227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:43.278920 1227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "c339e88fcb9a498e8bbea3a633e16253"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "2bdf6add00f261e180add87b2309586e"
server_key_iv: "a688be90a9e49d71ee618fbd32ffec95"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.279400 1227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:43.308944 1227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:43.311856 1227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:43.312017 1227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:43.312595 1227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:43.313586 1269 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:44:43.317363 1227 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:44:43.317412 1227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 18:44:43.317471 1227 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:44:43.318975 1227 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:44:43.319018 1227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 18:44:43.319260 1269 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Bootstrap starting.
I20260927 18:44:43.328016 1227 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:40299
I20260927 18:44:43.328102 1376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:40299 every 8 connection(s)
I20260927 18:44:43.328897 1227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:43.336836 1269 log.cc:824] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:43.336926 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1227
I20260927 18:44:43.337388 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:42179
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:32841
--webserver_port=34093
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:43.344683 1269 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:44:43.344961 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.331002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:38389 (local address 127.18.235.190:32841)
0927 18:44:43.331447 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.331465 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.331553 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:44:43.332645 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.332662 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.333067 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.333516 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:43.334428 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.336047 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.337191 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.337837 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.337983 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.340540 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:43.340545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:43.340556 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:43.340583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:43.342136 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.342610 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.342615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.342620 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.342733 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.343029 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.343032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.343034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.343353 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:43.343467 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.343962 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.344212 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":111,"threads_started":1}
I20260927 18:44:43.345077 1269 tablet_bootstrap.cc:492] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Bootstrap complete.
I20260927 18:44:43.345427 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.330937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:38389)
0927 18:44:43.331405 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.331454 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.332233 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.333512 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.333518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.333838 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:43.334266 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.334278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.336169 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.336171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.337062 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.337072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.337963 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.337967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.338145 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.338685 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:43.338706 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:43.340370 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:43.342253 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.342258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.342270 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.342502 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.342819 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.342822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.342824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.342945 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.343459 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:43.343464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:43.343689 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.343839 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.344138 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260927 18:44:43.346333 1269 ts_tablet_manager.cc:1397] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.015s
I20260927 18:44:43.347405 1377 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32841
I20260927 18:44:43.347604 1377 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:43.348006 1377 heartbeater.cc:511] Master 127.18.235.190:32841 requested a full tablet report, sending...
I20260927 18:44:43.349153 1269 raft_consensus.cc:348] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:43.349624 1269 raft_consensus.cc:729] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c339e88fcb9a498e8bbea3a633e16253, State: Initialized, Role: FOLLOWER
I20260927 18:44:43.350078 1269 consensus_queue.cc:260] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } }
I20260927 18:44:43.350263 1189 ts_manager.cc:194] Registered new tserver with Master: c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299)
I20260927 18:44:43.350952 1269 ts_tablet_manager.cc:1428] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 18:44:43.353406 1189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:38389
I20260927 18:44:43.355717 1377 heartbeater.cc:503] Master 127.18.235.190:32841 was elected leader, sending a full tablet report...
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:44:43.362669 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.352808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40299 (local address 127.0.0.1:41840)
0927 18:44:43.353467 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.353512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.353638 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.354091 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.354096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.354466 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:43.354575 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.354582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.355915 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.355919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.356893 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.356904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.357525 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.357529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.357642 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.358083 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:43.358097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:43.359409 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:43.361236 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.361243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.361247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.361443 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.361731 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.361734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.361736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.361810 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.362238 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:43.362242 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:43.362328 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.362427 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.362555 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":80,"threads_started":1}
I20260927 18:44:43.362810 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.352880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.18.235.129:40299)
0927 18:44:43.353229 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.353238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.353589 (+ 351us) server_negotiation.cc:407] Connection header received
0927 18:44:43.353776 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.353781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.353958 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.354091 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:43.354691 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.355713 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.357007 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.357419 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.357498 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.359532 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:43.359542 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:43.359554 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:43.359578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:43.361145 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.361534 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.361540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.361541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.361628 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.361883 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.361886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.361887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.362147 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:43.362265 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.362518 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.362665 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":93,"threads_started":1}
I20260927 18:44:43.363806 1311 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e281711175443f1b5fef2c66fdfff76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:42 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:41840
I20260927 18:44:43.364506 1386 tablet_replica.cc:330] stopping tablet replica
I20260927 18:44:43.364744 1386 raft_consensus.cc:2229] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:43.364934 1386 raft_consensus.cc:2258] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:43.365737 1386 ts_tablet_manager.cc:1905] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:43.368906 1386 ts_tablet_manager.cc:1918] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:43.368978 1386 log.cc:1196] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/1e281711175443f1b5fef2c66fdfff76
I20260927 18:44:43.369278 1386 ts_tablet_manager.cc:1939] T 1e281711175443f1b5fef2c66fdfff76 P c339e88fcb9a498e8bbea3a633e16253: Deleting consensus metadata
I20260927 18:44:43.370786 1175 catalog_manager.cc:4841] TS c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299): tablet 1e281711175443f1b5fef2c66fdfff76 (table test-table [id=f61783c9f56c4551a501dd06dfe69b8b]) successfully deleted
W20260927 18:44:43.447976 1382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:43.448206 1382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:43.448261 1382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:43.452128 1382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:43.452180 1382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:43.452255 1382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:44:43.457150 1382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:42179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=34093
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:43.458022 1382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:43.458921 1382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:43.467408 1391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.467504 1382 server_base.cc:1034] running on GCE node
W20260927 18:44:43.467443 1392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:43.467410 1394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.467962 1382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:43.468531 1382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:43.469676 1382 hybrid_clock.cc:648] HybridClock initialized: now 1790534683469653 us; error 38 us; skew 500 ppm
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:43.472077 1382 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:44:43.472841 1382 webserver.cc:460] Webserver started at http://127.18.235.130:34093/ using document root <none> and password file <none>
I20260927 18:44:43.473373 1382 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:43.473452 1382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:43.476841 1382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:43.478787 1401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.479859 1382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:44:43.479957 1382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "aeb505d6786f47108374a2782964e9a5"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "a62a9904b538b0a5b920d466496f473e"
server_key_iv: "58cfd41d450f05be3b974348c5732bfe"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.480427 1382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:43.496773 1382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:43.499272 1382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:43.499406 1382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:43.499871 1382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:43.500800 1408 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 18:44:43.504238 1408 ts_tablet_manager.cc:1852] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 18:44:43.504601 1408 ts_tablet_manager.cc:1905] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:43.508096 1408 ts_tablet_manager.cc:1918] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:43.508189 1408 log.cc:1196] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/1e281711175443f1b5fef2c66fdfff76
I20260927 18:44:43.508488 1408 ts_tablet_manager.cc:1939] T 1e281711175443f1b5fef2c66fdfff76 P aeb505d6786f47108374a2782964e9a5: Deleting consensus metadata
I20260927 18:44:43.508764 1382 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 18:44:43.508816 1382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260927 18:44:43.508858 1382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:43.508877 1382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.516675 1382 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42179
I20260927 18:44:43.516722 1515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42179 every 8 connection(s)
I20260927 18:44:43.517457 1382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:43.523581 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1382
I20260927 18:44:43.523959 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:43199
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:32841
--webserver_port=35341
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:43.529657 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.519013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:37911 (local address 127.18.235.190:32841)
0927 18:44:43.519142 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.519145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.519877 (+ 732us) server_negotiation.cc:407] Connection header received
0927 18:44:43.520700 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.520703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.520760 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.520828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:43.521904 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.522410 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.523457 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.523708 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.523799 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.526361 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:43.526365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:43.526368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:43.526395 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:43.527765 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.528237 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.528240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.528242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.528331 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.528597 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.528601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.528602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.528823 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:43.528924 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.529353 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.529481 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:44:43.530297 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.519343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:37911)
0927 18:44:43.519748 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.519787 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.520507 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.520994 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.521000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.521341 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:43.521768 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.521781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.522520 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.522524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.523339 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.523349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.523851 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.523855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.524039 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.524564 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:43.524582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:43.526200 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:43.527887 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.527893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.527906 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.528135 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.528411 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.528413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.528414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.528523 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.528909 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:43.528913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:43.529124 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.529284 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.529500 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260927 18:44:43.531307 1516 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32841
I20260927 18:44:43.531535 1516 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:43.532048 1516 heartbeater.cc:511] Master 127.18.235.190:32841 requested a full tablet report, sending...
I20260927 18:44:43.533037 1189 ts_manager.cc:194] Registered new tserver with Master: aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
I20260927 18:44:43.533578 1189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:37911
W20260927 18:44:43.628408 1519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:43.628623 1519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:43.628665 1519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:43.632560 1519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:43.632609 1519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:43.632673 1519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:44:43.637601 1519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:43199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=35341
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:43.638455 1519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:43.639336 1519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:43.647867 1525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:43.647877 1524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.647936 1519 server_base.cc:1034] running on GCE node
W20260927 18:44:43.647871 1527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.648445 1519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:43.648972 1519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:43.650118 1519 hybrid_clock.cc:648] HybridClock initialized: now 1790534683650109 us; error 55 us; skew 500 ppm
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:43.652412 1519 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:44:43.653178 1519 webserver.cc:460] Webserver started at http://127.18.235.131:35341/ using document root <none> and password file <none>
I20260927 18:44:43.653648 1519 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:43.653725 1519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:43.657073 1519 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:43.658881 1534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.659983 1519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:43.660094 1519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "1fef89d91c6a44048143ef9c4c02e873"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "3c8a97697d37faedba82c943d51929f7"
server_key_iv: "0ee3d3497fa988ef6917be14313270fb"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.660580 1519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:43.678419 1519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:43.680914 1519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:43.681046 1519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:43.681562 1519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:43.682310 1519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:43.682355 1519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.682405 1519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:43.682427 1519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.690590 1519 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:43199
I20260927 18:44:43.690618 1647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:43199 every 8 connection(s)
I20260927 18:44:43.691461 1519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:43.699908 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1519
Sep 27 18:44:43 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:44:43.703836 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.693014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:33809 (local address 127.18.235.190:32841)
0927 18:44:43.693161 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.693166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.693928 (+ 762us) server_negotiation.cc:407] Connection header received
0927 18:44:43.695265 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.695268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.695324 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.695394 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:43.696502 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.697011 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.698143 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.698358 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.698444 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.700607 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:43.700611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:43.700613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:43.700641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:43.702051 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.702467 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.702470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.702471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.702569 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.702863 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.702866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.702868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.703036 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:43.703117 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.703570 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.703684 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:44:43.704725 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.693393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:33809)
0927 18:44:43.693785 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.693824 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.695070 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.695573 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.695580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.695918 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:43.696355 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.696368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.697144 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.697148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.698028 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.698037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.698486 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.698491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.698643 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.699127 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:43.699142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:43.700459 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:43.702148 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.702153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.702162 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.702389 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.702641 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:43.702645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:43.702647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:43.702776 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:43.703131 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:43.703138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:43.703348 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.703502 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.703779 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260927 18:44:43.705856 1648 heartbeater.cc:347] Connected to a master server at 127.18.235.190:32841
I20260927 18:44:43.706072 1648 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:43.706506 1648 heartbeater.cc:511] Master 127.18.235.190:32841 requested a full tablet report, sending...
I20260927 18:44:43.707497 1189 ts_manager.cc:194] Registered new tserver with Master: 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199)
I20260927 18:44:43.707950 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.700488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.18.235.190:32841)
0927 18:44:43.700690 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.700694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.700850 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:44:43.700958 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.700960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.701019 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.701122 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:43.701897 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.702500 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.703313 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.703540 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.703623 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.705019 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:43.705023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:43.705025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:43.705049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:43.706475 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.706888 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.706890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.706891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.706958 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.707207 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.707210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.707212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.707362 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:43.707554 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.707696 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.707806 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260927 18:44:43.708153 1189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:33809
I20260927 18:44:43.709492 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:44:43.712742 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.709970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.18.235.190:32841)
0927 18:44:43.710073 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.710077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.710234 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:44:43.710324 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.710326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.710399 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.710469 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:43.710596 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.711045 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.711753 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.711947 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.712012 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.712196 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:43.712498 (+ 302us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:43.712577 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.712603 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.712642 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:44:43.713270 1189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:43.716181 1189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:43.722690 1311 tablet_service.cc:1467] Processing CreateTablet for tablet 42b3096a21f34317a6ddca2b84bf2d2f (DEFAULT_TABLE table=test-table [id=aa3263a50095462db0cdb4d08ef2488f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:43.723286 1311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b3096a21f34317a6ddca2b84bf2d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:43.725955 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.721666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:43199 (local address 127.0.0.1:44426)
0927 18:44:43.721792 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.721804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.721862 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.722596 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.722599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.722616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:43.722697 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.722703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.723737 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.723740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.724752 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.724761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.725514 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.725517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.725666 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.725703 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.725771 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.725842 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":38}
I20260927 18:44:43.725996 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.721807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.18.235.131:43199)
0927 18:44:43.722146 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.722157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.722176 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:44:43.722234 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.722239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.722420 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.722611 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:43.722779 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.723632 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.724856 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.725413 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.725521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.725636 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.725807 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.725886 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260927 18:44:43.725992 1269 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Bootstrap starting.
I20260927 18:44:43.726778 1269 tablet_bootstrap.cc:654] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:43.727077 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.721904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42179 (local address 127.0.0.1:51000)
0927 18:44:43.722384 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.722397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.722476 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.723150 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.723153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.723171 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:43.723243 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.723249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.724546 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.724549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.725715 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.725726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.726716 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.726720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.726804 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.726825 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.726882 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.726945 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":59,"thread_start_us":114,"threads_started":1}
I20260927 18:44:43.727152 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.722307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.18.235.130:42179)
0927 18:44:43.722728 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.722739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.722761 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:44:43.722832 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.722837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.723014 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.723161 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:43.723357 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.724382 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.725843 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.726610 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.726729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.726849 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.726909 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.726988 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":82,"threads_started":1}
I20260927 18:44:43.728201 1269 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: No bootstrap required, opened a new log
I20260927 18:44:43.728291 1269 ts_tablet_manager.cc:1397] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 18:44:43.728691 1269 raft_consensus.cc:348] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.728809 1269 raft_consensus.cc:374] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:43.728852 1269 raft_consensus.cc:729] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c339e88fcb9a498e8bbea3a633e16253, State: Initialized, Role: FOLLOWER
I20260927 18:44:43.728962 1269 consensus_queue.cc:260] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.728746 1450 tablet_service.cc:1467] Processing CreateTablet for tablet 42b3096a21f34317a6ddca2b84bf2d2f (DEFAULT_TABLE table=test-table [id=aa3263a50095462db0cdb4d08ef2488f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:43.729239 1269 ts_tablet_manager.cc:1428] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:44:43.728945 1582 tablet_service.cc:1467] Processing CreateTablet for tablet 42b3096a21f34317a6ddca2b84bf2d2f (DEFAULT_TABLE table=test-table [id=aa3263a50095462db0cdb4d08ef2488f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:43.729733 1450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b3096a21f34317a6ddca2b84bf2d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:43.729862 1582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b3096a21f34317a6ddca2b84bf2d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:43.733819 1408 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Bootstrap starting.
I20260927 18:44:43.735525 1408 tablet_bootstrap.cc:654] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:43.736166 1661 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Bootstrap starting.
I20260927 18:44:43.736339 1408 log.cc:824] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:43.738173 1661 tablet_bootstrap.cc:654] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:43.738323 1408 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: No bootstrap required, opened a new log
I20260927 18:44:43.738507 1408 ts_tablet_manager.cc:1397] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:44:43.739310 1661 log.cc:824] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:43.741060 1661 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: No bootstrap required, opened a new log
I20260927 18:44:43.741259 1661 ts_tablet_manager.cc:1397] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260927 18:44:43.742034 1408 raft_consensus.cc:348] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.742273 1408 raft_consensus.cc:374] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:43.742326 1408 raft_consensus.cc:729] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb505d6786f47108374a2782964e9a5, State: Initialized, Role: FOLLOWER
I20260927 18:44:43.742750 1408 consensus_queue.cc:260] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.743319 1516 heartbeater.cc:503] Master 127.18.235.190:32841 was elected leader, sending a full tablet report...
I20260927 18:44:43.743682 1408 ts_tablet_manager.cc:1428] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:44:43.744544 1661 raft_consensus.cc:348] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.744717 1661 raft_consensus.cc:374] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:43.744764 1661 raft_consensus.cc:729] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fef89d91c6a44048143ef9c4c02e873, State: Initialized, Role: FOLLOWER
I20260927 18:44:43.745165 1661 consensus_queue.cc:260] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.745709 1648 heartbeater.cc:503] Master 127.18.235.190:32841 was elected leader, sending a full tablet report...
I20260927 18:44:43.746026 1661 ts_tablet_manager.cc:1428] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
W20260927 18:44:43.768196 1517 tablet.cc:2369] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:43.830061 1378 tablet.cc:2369] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:43.942426 1649 tablet.cc:2369] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:43.972297 1664 raft_consensus.cc:482] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:43.972469 1664 raft_consensus.cc:504] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.973635 1664 leader_election.cc:288] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199), c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299)
I20260927 18:44:43.977762 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.973836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:43199 (local address 127.18.235.130:45117)
0927 18:44:43.973937 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.973950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.974046 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.974318 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.974321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.974334 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:43.974448 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.974453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.975374 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.975376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.976587 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.976597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.977389 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.977391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.977471 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.977491 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.977552 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.977624 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 18:44:43.977764 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.973906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:45117 (local address 127.18.235.131:43199)
0927 18:44:43.974017 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.974021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.974038 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:44:43.974169 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.974172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.974221 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.974292 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:43.974551 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.975274 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.976699 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.977295 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.977407 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.977457 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.977584 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.977650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:44:43.977958 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.973950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:41591 (local address 127.18.235.129:40299)
0927 18:44:43.974252 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.974256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.974272 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:43.974415 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.974417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.974468 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.974567 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:43.974800 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.975480 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.976803 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.977486 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.977589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.977722 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.977774 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.977842 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20260927 18:44:43.978093 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.973843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40299 (local address 127.18.235.130:41591)
0927 18:44:43.974218 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:43.974228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:43.974311 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:43.974592 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:43.974594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:43.974609 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:43.974704 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.974708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.975581 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.975584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.976702 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:43.976710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.977589 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.977592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:43.977678 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.977699 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:44:43.977748 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:44:43.977917 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":92,"thread_start_us":87,"threads_started":1}
I20260927 18:44:43.978403 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b3096a21f34317a6ddca2b84bf2d2f" candidate_uuid: "aeb505d6786f47108374a2782964e9a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fef89d91c6a44048143ef9c4c02e873" is_pre_election: true
I20260927 18:44:43.978693 1602 raft_consensus.cc:2454] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb505d6786f47108374a2782964e9a5 in term 0.
I20260927 18:44:43.978695 1331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b3096a21f34317a6ddca2b84bf2d2f" candidate_uuid: "aeb505d6786f47108374a2782964e9a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c339e88fcb9a498e8bbea3a633e16253" is_pre_election: true
I20260927 18:44:43.978926 1331 raft_consensus.cc:2454] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb505d6786f47108374a2782964e9a5 in term 0.
I20260927 18:44:43.979121 1405 leader_election.cc:302] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fef89d91c6a44048143ef9c4c02e873, aeb505d6786f47108374a2782964e9a5; no voters:
I20260927 18:44:43.979389 1664 raft_consensus.cc:2781] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:43.979462 1664 raft_consensus.cc:482] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:43.979490 1664 raft_consensus.cc:3037] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:43.980450 1664 raft_consensus.cc:504] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.980743 1664 leader_election.cc:288] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [CANDIDATE]: Term 1 election: Requested vote from peers 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199), c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299)
I20260927 18:44:43.981024 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b3096a21f34317a6ddca2b84bf2d2f" candidate_uuid: "aeb505d6786f47108374a2782964e9a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fef89d91c6a44048143ef9c4c02e873"
I20260927 18:44:43.981045 1331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b3096a21f34317a6ddca2b84bf2d2f" candidate_uuid: "aeb505d6786f47108374a2782964e9a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c339e88fcb9a498e8bbea3a633e16253"
I20260927 18:44:43.981145 1602 raft_consensus.cc:3037] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:43.981199 1331 raft_consensus.cc:3037] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:43.982106 1331 raft_consensus.cc:2454] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb505d6786f47108374a2782964e9a5 in term 1.
I20260927 18:44:43.982144 1602 raft_consensus.cc:2454] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb505d6786f47108374a2782964e9a5 in term 1.
I20260927 18:44:43.982349 1404 leader_election.cc:302] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb505d6786f47108374a2782964e9a5, c339e88fcb9a498e8bbea3a633e16253; no voters:
I20260927 18:44:43.982496 1664 raft_consensus.cc:2781] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:43.982699 1664 raft_consensus.cc:686] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 1 LEADER]: Becoming Leader. State: Replica: aeb505d6786f47108374a2782964e9a5, State: Running, Role: LEADER
I20260927 18:44:43.982995 1664 consensus_queue.cc:237] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:43.985807 1189 catalog_manager.cc:5495] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb505d6786f47108374a2782964e9a5 (127.18.235.130). New cstate: current_term: 1 leader_uuid: "aeb505d6786f47108374a2782964e9a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } health_report { overall_health: UNKNOWN } } }
I20260927 18:44:44.016542 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.013527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.18.235.130:42179)
0927 18:44:44.013645 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:44.013649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:44.013661 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:44:44.013796 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:44.013798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:44.013846 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:44.013923 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:44.014078 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.014541 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.015284 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.015510 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.015584 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.015829 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:44.016200 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:44.016286 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:44.016331 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 18:44:44.016380 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:44:44.021632 1331 raft_consensus.cc:1261] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Refusing update from remote peer aeb505d6786f47108374a2782964e9a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:44.021636 1602 raft_consensus.cc:1261] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Refusing update from remote peer aeb505d6786f47108374a2782964e9a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:44.022183 1668 consensus_queue.cc:1035] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:44.022334 1664 consensus_queue.cc:1035] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:44.026774 1671 mvcc.cc:205] Tried to move back new op lower bound from 7334030065746767872 to 7334030065596936192. Current Snapshot: MvccSnapshot[applied={T|T < 7334030065746767872}]
I20260927 18:44:44.036019 1189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:44.036140 1189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:44.038172 1189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Sending DeleteTablet for 3 replicas of tablet 42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:44.038729 1582 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b3096a21f34317a6ddca2b84bf2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:44 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:44426
I20260927 18:44:44.038859 1311 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b3096a21f34317a6ddca2b84bf2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:44 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:41840
I20260927 18:44:44.038906 1450 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b3096a21f34317a6ddca2b84bf2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:44 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:51000
I20260927 18:44:44.039157 1681 tablet_replica.cc:330] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: stopping tablet replica
I20260927 18:44:44.039297 1682 tablet_replica.cc:330] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: stopping tablet replica
I20260927 18:44:44.039314 1681 raft_consensus.cc:2229] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:44.039424 1681 raft_consensus.cc:2258] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:44.039508 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 1227
I20260927 18:44:44.039551 1682 raft_consensus.cc:2229] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:44.039943 1682 raft_consensus.cc:2258] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:44.041266 1682 ts_tablet_manager.cc:1905] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:44.043795 1682 ts_tablet_manager.cc:1918] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:44.043884 1682 log.cc:1196] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:44.043882 1680 tablet_replica.cc:330] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: stopping tablet replica
I20260927 18:44:44.044137 1682 ts_tablet_manager.cc:1939] T 42b3096a21f34317a6ddca2b84bf2d2f P aeb505d6786f47108374a2782964e9a5: Deleting consensus metadata
I20260927 18:44:44.044133 1680 raft_consensus.cc:2229] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:44.044394 1680 raft_consensus.cc:2258] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:44.045306 1174 catalog_manager.cc:4841] TS aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179): tablet 42b3096a21f34317a6ddca2b84bf2d2f (table test-table [id=aa3263a50095462db0cdb4d08ef2488f]) successfully deleted
I20260927 18:44:44.045883 1680 ts_tablet_manager.cc:1905] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:44.048276 1680 ts_tablet_manager.cc:1918] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:44.048359 1680 log.cc:1196] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:44.048657 1680 ts_tablet_manager.cc:1939] T 42b3096a21f34317a6ddca2b84bf2d2f P 1fef89d91c6a44048143ef9c4c02e873: Deleting consensus metadata
W20260927 18:44:44.048980 1175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:40299 (error 108)
I20260927 18:44:44.049263 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 1382
I20260927 18:44:44.050293 1176 catalog_manager.cc:4841] TS 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199): tablet 42b3096a21f34317a6ddca2b84bf2d2f (table test-table [id=aa3263a50095462db0cdb4d08ef2488f]) successfully deleted
I20260927 18:44:44.053893 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.053607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:40299 (local address 127.0.0.1:41854)
0927 18:44:44.053761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.053825 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:40299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 18:44:44.054036 1175 catalog_manager.cc:4584] TS c339e88fcb9a498e8bbea3a633e16253 (127.18.235.129:40299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42b3096a21f34317a6ddca2b84bf2d2f: Network error: Client connection negotiation failed: client connection to 127.18.235.129:40299: connect: Connection refused (error 111)
I20260927 18:44:44.054620 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 1519
I20260927 18:44:44.058842 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 1156
I20260927 18:44:44.063807 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32841
--webserver_interface=127.18.235.190
--webserver_port=37173
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:44.165247 1684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:44.165544 1684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:44.165616 1684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:44.169525 1684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:44.169572 1684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:44.169590 1684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:44.169605 1684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:44.169617 1684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:44.174505 1684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=37173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:44.175390 1684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:44.176277 1684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:44.183878 1690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.183897 1692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.183897 1689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.184054 1684 server_base.cc:1034] running on GCE node
I20260927 18:44:44.184599 1684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:44.185595 1684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:44.186777 1684 hybrid_clock.cc:648] HybridClock initialized: now 1790534684186751 us; error 34 us; skew 500 ppm
Sep 27 18:44:44 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:44.189229 1684 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:44:44.190002 1684 webserver.cc:460] Webserver started at http://127.18.235.190:37173/ using document root <none> and password file <none>
I20260927 18:44:44.190492 1684 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:44.190555 1684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:44.194104 1684 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:44.196012 1699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.197129 1684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:44:44.197255 1684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "0f87b9ab28c34c619ba5ab5feebfa920"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "b3dfc4c77065ba1d8815676bc48a8fd4"
server_key_iv: "f5d7ac91abd9419483090dcd66c9b5dc"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.197753 1684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:44.222050 1684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:44.224467 1684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:44.224622 1684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:44.230855 1684 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:32841
I20260927 18:44:44.231081 1751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:32841 every 8 connection(s)
I20260927 18:44:44.231828 1684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:44.235677 1752 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:44:44.237191 1752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap starting.
I20260927 18:44:44.239142 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1684
I20260927 18:44:44.239548 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:40299
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:32841
--webserver_port=36391
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:44.253531 1752 log.cc:824] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:44.263737 1752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:44:44.264020 1752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920: Bootstrap complete.
I20260927 18:44:44.267638 1752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:44.267974 1752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f87b9ab28c34c619ba5ab5feebfa920, State: Initialized, Role: FOLLOWER
I20260927 18:44:44.268393 1752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:44.268507 1752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:44.268567 1752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:44.268651 1752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:44:44.270346 1752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:44.270632 1752 leader_election.cc:302] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f87b9ab28c34c619ba5ab5feebfa920; no voters:
I20260927 18:44:44.270877 1752 leader_election.cc:288] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:44:44.270958 1756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:44:44.271168 1756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [term 3 LEADER]: Becoming Leader. State: Replica: 0f87b9ab28c34c619ba5ab5feebfa920, State: Running, Role: LEADER
I20260927 18:44:44.271440 1756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } }
I20260927 18:44:44.272006 1752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:44.272298 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f87b9ab28c34c619ba5ab5feebfa920. Latest consensus state: current_term: 3 leader_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } } }
I20260927 18:44:44.272485 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:44.272701 1757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f87b9ab28c34c619ba5ab5feebfa920" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 32841 } } }
I20260927 18:44:44.272840 1757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f87b9ab28c34c619ba5ab5feebfa920 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:44.273602 1760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:44.278096 1760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:44.278709 1760 catalog_manager.cc:1259] Loaded cluster ID: d0f6da3ebe4d4600b9e7fd51e2e5df0f
I20260927 18:44:44.278765 1760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 18:44:44.279717 1760 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6fca3369ad google::LogMessage::Fail() at ??:0
@ 0x7f6fca33988a google::LogMessage::SendToLog() at ??:0
@ 0x7f6fca3364de google::LogMessage::Flush() at ??:0
@ 0x7f6fca337a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6fd4c75fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6fd4c76d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6fd4c73579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6fd4cb7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6fcc5883d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6fcc588c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6fcc58a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6fcc579512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6fcdbbd6db start_thread at ??:0
@ 0x7f6fc8e3461f clone at ??:0
W20260927 18:44:44.369397 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:44.369704 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:44.369772 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:44.375572 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:44.375646 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:44.375767 1754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:44:44.383118 1754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:40299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=36391
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:44.384315 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:44.385541 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:44.396955 1796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.397295 1793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.397465 1794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.397470 1754 server_base.cc:1034] running on GCE node
I20260927 18:44:44.397928 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:44.398586 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:44.399740 1754 hybrid_clock.cc:648] HybridClock initialized: now 1790534684399712 us; error 37 us; skew 500 ppm
Sep 27 18:44:44 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:44.402791 1754 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:44:44.403812 1754 webserver.cc:460] Webserver started at http://127.18.235.129:36391/ using document root <none> and password file <none>
I20260927 18:44:44.404479 1754 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:44.404565 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:44.409288 1754 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:44.411739 1803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.413060 1754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:44.413188 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "c339e88fcb9a498e8bbea3a633e16253"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "2bdf6add00f261e180add87b2309586e"
server_key_iv: "a688be90a9e49d71ee618fbd32ffec95"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.413739 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:44.443230 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:44.446108 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:44.446254 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:44.446760 1754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:44.447767 1810 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:44:44.452967 1754 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:44:44.453030 1754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 18:44:44.453086 1754 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:44:44.454555 1754 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:44:44.454596 1754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 18:44:44.454919 1810 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Bootstrap starting.
I20260927 18:44:44.463666 1754 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:40299
I20260927 18:44:44.463914 1917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:40299 every 8 connection(s)
I20260927 18:44:44.464542 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:44.465660 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1754
I20260927 18:44:44.466099 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:42179
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:32841
--webserver_port=34093
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:44.474948 1810 log.cc:824] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:44.482968 1810 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:44:44.483341 1810 tablet_bootstrap.cc:492] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Bootstrap complete.
I20260927 18:44:44.484575 1810 ts_tablet_manager.cc:1397] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260927 18:44:44.487524 1810 raft_consensus.cc:348] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:44.487999 1810 raft_consensus.cc:729] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c339e88fcb9a498e8bbea3a633e16253, State: Initialized, Role: FOLLOWER
I20260927 18:44:44.488474 1810 consensus_queue.cc:260] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:44.489409 1810 ts_tablet_manager.cc:1428] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260927 18:44:44.512465 1921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:44.468851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:39553)
0927 18:44:44.469392 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.469448 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:44.470454 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:44.511776 (+ 41322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":155,"threads_started":1}
W20260927 18:44:44.513055 1918 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:32841 (0 consecutive failures): Network error: Failed to ping master at 127.18.235.190:32841: Client connection negotiation failed: client connection to 127.18.235.190:32841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:44:44.513916 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.513635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:47419)
0927 18:44:44.513772 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.513847 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 18:44:44.514546 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.514336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:55507)
0927 18:44:44.514432 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.514480 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:44:44.514698 1918 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:44:44.577247 1920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:44.577454 1920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:44.577492 1920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:44.581254 1920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:44.581292 1920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:44.581344 1920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:44:44.586193 1920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:42179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=34093
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:44.586983 1920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:44.587808 1920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:44.596165 1928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.596155 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.596199 1920 server_base.cc:1034] running on GCE node
W20260927 18:44:44.596156 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.596678 1920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:44.597250 1920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:44.598393 1920 hybrid_clock.cc:648] HybridClock initialized: now 1790534684598369 us; error 36 us; skew 500 ppm
Sep 27 18:44:44 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:44.600735 1920 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:44:44.601473 1920 webserver.cc:460] Webserver started at http://127.18.235.130:34093/ using document root <none> and password file <none>
I20260927 18:44:44.601934 1920 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:44.601989 1920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:44.605273 1920 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:44.607023 1938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.608031 1920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:44.608120 1920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "aeb505d6786f47108374a2782964e9a5"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "a62a9904b538b0a5b920d466496f473e"
server_key_iv: "58cfd41d450f05be3b974348c5732bfe"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.608569 1920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:44.635900 1920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:44.638525 1920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:44.638674 1920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:44.639143 1920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:44.639810 1920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:44.639853 1920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.639886 1920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:44.639906 1920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.647643 1920 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42179
I20260927 18:44:44.647672 2051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42179 every 8 connection(s)
I20260927 18:44:44.648555 1920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:44.651891 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.650366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:52005)
0927 18:44:44.650775 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.650871 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
W20260927 18:44:44.652256 2052 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:32841 (0 consecutive failures): Network error: Failed to ping master at 127.18.235.190:32841: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
I20260927 18:44:44.652734 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.652559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:36503)
0927 18:44:44.652646 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.652688 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 18:44:44.653272 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.653087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:53717)
0927 18:44:44.653194 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.653230 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 18:44:44.653436 2052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:44:44.653816 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 1920
I20260927 18:44:44.654163 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:43199
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:32841
--webserver_port=35341
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:44.757220 2055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:44.757427 2055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:44.757468 2055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:44.761269 2055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:44.761317 2055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:44.761373 2055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:44:44.766162 2055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:43199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=35341
--tserver_master_addrs=127.18.235.190:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:44:44.766989 2055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:44.767810 2055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:44.776295 2061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.776327 2055 server_base.cc:1034] running on GCE node
W20260927 18:44:44.776266 2063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.776473 2060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.776822 2055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:44.777381 2055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:44.778532 2055 hybrid_clock.cc:648] HybridClock initialized: now 1790534684778518 us; error 26 us; skew 500 ppm
Sep 27 18:44:44 dist-test-slave-6lr6 krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:44.780869 2055 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:44:44.781612 2055 webserver.cc:460] Webserver started at http://127.18.235.131:35341/ using document root <none> and password file <none>
I20260927 18:44:44.782064 2055 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:44.782124 2055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:44.785370 2055 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:44.787081 2070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.788080 2055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:44.788173 2055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "1fef89d91c6a44048143ef9c4c02e873"
format_stamp: "Formatted at 2026-09-27 18:44:42 on dist-test-slave-6lr6"
server_key: "3c8a97697d37faedba82c943d51929f7"
server_key_iv: "0ee3d3497fa988ef6917be14313270fb"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.788656 2055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:44.802506 2055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:44.805052 2055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:44.805214 2055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:44.805680 2055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:44.806347 2055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:44.806391 2055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.806435 2055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:44.806461 2055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.814061 2055 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:43199
I20260927 18:44:44.814093 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:43199 every 8 connection(s)
I20260927 18:44:44.814857 2055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:44.818176 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.816613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:49789)
0927 18:44:44.816994 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.817125 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
W20260927 18:44:44.818544 2184 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:32841 (0 consecutive failures): Network error: Failed to ping master at 127.18.235.190:32841: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
I20260927 18:44:44.819031 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2055
I20260927 18:44:44.819052 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.818873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:59639)
0927 18:44:44.818960 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.819002 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:44:44.819598 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.819408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:34277)
0927 18:44:44.819499 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.819554 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:44:44.819729 2184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:44:45.515650 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.515196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:50431)
0927 18:44:45.515491 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.515546 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":77,"threads_started":1}
I20260927 18:44:45.654191 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.653851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:54643)
0927 18:44:45.654056 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.654102 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
I20260927 18:44:45.820518 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.820141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:32971)
0927 18:44:45.820372 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.820431 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260927 18:44:46.022493 2191 raft_consensus.cc:482] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:46.022665 2191 raft_consensus.cc:504] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:46.023630 2191 leader_election.cc:288] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199), aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
Sep 27 18:44:46 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.130@KRBTEST.COM
Sep 27 18:44:46 dist-test-slave-6lr6 krb5kdc[608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.131@KRBTEST.COM
I20260927 18:44:46.047489 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.023833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42179 (local address 127.18.235.129:56075)
0927 18:44:46.024176 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:46.024189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:46.024276 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:46.026537 (+ 2261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:46.026539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:46.026700 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:46.026954 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:46.026970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.028165 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.028168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:46.029091 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:46.029121 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.029754 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.029757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:46.029923 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.033555 (+ 3632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:46.033607 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:46.039911 (+ 6304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:46.044737 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:46.044743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:46.044756 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:46.045087 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:46.045480 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:46.045485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:46.045487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:46.045663 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:46.046437 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:46.046443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:46.046753 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 18:44:46.047000 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 18:44:46.047304 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":104,"spinlock_wait_cycles":48512,"thread_start_us":66,"threads_started":1}
I20260927 18:44:46.047592 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.024029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:56075 (local address 127.18.235.130:42179)
0927 18:44:46.024397 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:46.024415 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:46.024444 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:44:46.025309 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:46.025318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:46.025760 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:46.026083 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:46.027106 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.028040 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.029218 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.029648 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.029794 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.040651 (+ 10857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:46.040658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:46.040676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:46.040710 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:46.044630 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:46.045229 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:46.045235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:46.045241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:46.045396 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:46.045771 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:46.045775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:46.045777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:46.046326 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:46.046463 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:46.047125 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 18:44:46.047435 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":78,"threads_started":1}
I20260927 18:44:46.048497 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.023764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:43199 (local address 127.18.235.129:51117)
0927 18:44:46.024049 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:46.024063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:46.024162 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:46.026131 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:46.026136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:46.026482 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:46.026955 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:46.026971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.028167 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.028170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:46.029088 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:46.029097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.029866 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.029870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:46.029952 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.033610 (+ 3658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:46.033626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:46.040278 (+ 6652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:46.046211 (+ 5933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:46.046215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:46.046218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:46.046467 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:46.046847 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:46.046850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:46.046852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:46.046939 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:46.047998 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:46.048001 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:46.048078 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 18:44:46.048166 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:44:46.048294 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":100,"spinlock_wait_cycles":896,"thread_start_us":65,"threads_started":1}
I20260927 18:44:46.048732 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.023930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:51117 (local address 127.18.235.131:43199)
0927 18:44:46.024348 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:46.024372 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:46.024418 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:44:46.025309 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:46.025318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:46.025760 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:46.026083 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:46.027109 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.028042 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.029203 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.029664 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.029803 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.040652 (+ 10849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:46.040660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:46.040676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:46.040709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:46.046110 (+ 5401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:46.046611 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:46.046616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:46.046621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:46.046765 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:46.047039 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:46.047043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:46.047044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:46.047452 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:46.047575 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:46.048305 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 18:44:46.048570 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":83,"threads_started":1}
W20260927 18:44:46.050374 1805 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
W20260927 18:44:46.051100 1807 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:46.051183 1807 leader_election.cc:302] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c339e88fcb9a498e8bbea3a633e16253; no voters: 1fef89d91c6a44048143ef9c4c02e873, aeb505d6786f47108374a2782964e9a5
I20260927 18:44:46.051470 2191 raft_consensus.cc:2726] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:44:46.516533 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.516279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:55657)
0927 18:44:46.516403 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:46.516460 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 18:44:46.655144 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.654787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:39301)
0927 18:44:46.655013 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:46.655062 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260927 18:44:46.821480 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.821156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:50107)
0927 18:44:46.821358 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:46.821404 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260927 18:44:47.517495 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:47.517139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:48507)
0927 18:44:47.517359 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:47.517407 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260927 18:44:47.656055 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:47.655728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:44711)
0927 18:44:47.655935 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:47.655979 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260927 18:44:47.822366 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:47.822013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:49141)
0927 18:44:47.822241 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:47.822289 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
I20260927 18:44:48.192421 2201 raft_consensus.cc:482] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:48.192543 2201 raft_consensus.cc:504] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:48.192936 2201 leader_election.cc:288] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199), aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
W20260927 18:44:48.193796 1805 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
W20260927 18:44:48.193909 1807 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:48.193988 1807 leader_election.cc:302] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c339e88fcb9a498e8bbea3a633e16253; no voters: 1fef89d91c6a44048143ef9c4c02e873, aeb505d6786f47108374a2782964e9a5
I20260927 18:44:48.194140 2201 raft_consensus.cc:2726] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:44:48.518445 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:48.518088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:59849)
0927 18:44:48.518308 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:48.518358 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260927 18:44:48.656944 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:48.656601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:35425)
0927 18:44:48.656813 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:48.656863 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260927 18:44:48.823400 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:48.823002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:56179)
0927 18:44:48.823255 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:48.823315 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260927 18:44:49.519547 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:49.519147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:59825)
0927 18:44:49.519409 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:49.519458 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260927 18:44:49.657895 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:49.657551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:33579)
0927 18:44:49.657764 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:49.657813 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260927 18:44:49.824429 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:49.824036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:45795)
0927 18:44:49.824287 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:49.824341 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260927 18:44:50.036449 2208 raft_consensus.cc:482] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:50.036562 2208 raft_consensus.cc:504] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:50.036911 2208 leader_election.cc:288] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199), aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
W20260927 18:44:50.037633 1805 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
W20260927 18:44:50.037747 1807 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:50.037837 1807 leader_election.cc:302] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c339e88fcb9a498e8bbea3a633e16253; no voters: 1fef89d91c6a44048143ef9c4c02e873, aeb505d6786f47108374a2782964e9a5
I20260927 18:44:50.037981 2208 raft_consensus.cc:2726] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:44:50.520556 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:50.520134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:39435)
0927 18:44:50.520387 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:50.520449 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260927 18:44:50.658778 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:50.658450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:50433)
0927 18:44:50.658654 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:50.658701 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260927 18:44:50.825426 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:50.825047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:50619)
0927 18:44:50.825292 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:50.825344 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260927 18:44:51.521612 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:51.521255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:43451)
0927 18:44:51.521473 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:51.521522 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260927 18:44:51.659715 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:51.659343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:53579)
0927 18:44:51.659588 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:51.659636 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":62,"threads_started":1}
I20260927 18:44:51.826411 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:51.826014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:39379)
0927 18:44:51.826278 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:51.826328 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260927 18:44:52.522533 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:52.522182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:60993)
0927 18:44:52.522396 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:52.522444 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260927 18:44:52.660640 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:52.660291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:46427)
0927 18:44:52.660504 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:52.660553 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260927 18:44:52.827354 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:52.826999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:36901)
0927 18:44:52.827212 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:52.827274 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260927 18:44:53.523512 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.523140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:43383)
0927 18:44:53.523375 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:53.523422 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260927 18:44:53.661593 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.661242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:49683)
0927 18:44:53.661459 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:53.661509 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260927 18:44:53.828271 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.827947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:45323)
0927 18:44:53.828147 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:53.828194 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260927 18:44:54.501515 2221 raft_consensus.cc:482] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:54.501645 2221 raft_consensus.cc:504] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fef89d91c6a44048143ef9c4c02e873" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 43199 } } peers { permanent_uuid: "aeb505d6786f47108374a2782964e9a5" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42179 } } peers { permanent_uuid: "c339e88fcb9a498e8bbea3a633e16253" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 40299 } }
I20260927 18:44:54.501986 2221 leader_election.cc:288] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199), aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179)
W20260927 18:44:54.502661 1805 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aeb505d6786f47108374a2782964e9a5 (127.18.235.130:42179): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
W20260927 18:44:54.502774 1807 leader_election.cc:341] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fef89d91c6a44048143ef9c4c02e873 (127.18.235.131:43199): Not found: Tablet not found: 42b3096a21f34317a6ddca2b84bf2d2f
I20260927 18:44:54.502857 1807 leader_election.cc:302] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c339e88fcb9a498e8bbea3a633e16253; no voters: 1fef89d91c6a44048143ef9c4c02e873, aeb505d6786f47108374a2782964e9a5
I20260927 18:44:54.502987 2221 raft_consensus.cc:2726] T 42b3096a21f34317a6ddca2b84bf2d2f P c339e88fcb9a498e8bbea3a633e16253 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:44:54.524503 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:54.524164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:46473)
0927 18:44:54.524373 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:54.524422 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260927 18:44:54.662506 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:54.662163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:47619)
0927 18:44:54.662377 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:54.662425 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260927 18:44:54.829277 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:54.828879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:40921)
0927 18:44:54.829130 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:54.829184 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260927 18:44:55.525507 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:55.525059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:55945)
0927 18:44:55.525332 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:55.525393 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260927 18:44:55.663468 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:55.663101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:49309)
0927 18:44:55.663335 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:55.663382 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260927 18:44:55.830338 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:55.829944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:36713)
0927 18:44:55.830179 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:55.830241 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260927 18:44:56.526485 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:56.526112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:55471)
0927 18:44:56.526341 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:56.526393 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260927 18:44:56.664379 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:56.664053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:49777)
0927 18:44:56.664254 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:56.664303 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260927 18:44:56.831300 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:56.830928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:40041)
0927 18:44:56.831155 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:56.831205 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260927 18:44:57.527459 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:57.527102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:35619)
0927 18:44:57.527316 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:57.527365 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260927 18:44:57.665359 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:57.664980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:34493)
0927 18:44:57.665225 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:57.665276 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260927 18:44:57.832212 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:57.831874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:35129)
0927 18:44:57.832082 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:57.832132 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260927 18:44:58.528461 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.528059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:60955)
0927 18:44:58.528320 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:58.528372 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
I20260927 18:44:58.666333 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.665953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:47515)
0927 18:44:58.666186 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:58.666238 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260927 18:44:58.833209 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.832811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:37871)
0927 18:44:58.833050 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:58.833123 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260927 18:44:59.529603 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.529256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.129:41009)
0927 18:44:59.529464 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:59.529515 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260927 18:44:59.667357 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.667023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.130:52907)
0927 18:44:59.667220 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:59.667268 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260927 18:44:59.828516 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 1754
I20260927 18:44:59.833297 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 1920
I20260927 18:44:59.834640 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.834176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:32841 (local address 127.18.235.131:35171)
0927 18:44:59.834480 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:59.834531 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":60,"threads_started":1}
I20260927 18:44:59.837764 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2055
2026-09-27T18:44:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18918 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:59 dist-test-slave-6lr6 krb5kdc[2242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:59 dist-test-slave-6lr6 krb5kdc[2242](info): set up 2 sockets
Sep 27 18:44:59 dist-test-slave-6lr6 krb5kdc[2242](info): commencing operation
krb5kdc: starting...
W20260927 18:45:00.813093 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:45:00 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:45:00.918049 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44351
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:01.020630 2255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:01.020840 2255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:01.020881 2255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:01.024782 2255 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:45:01.024827 2255 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:45:01.024840 2255 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:45:01.024865 2255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:01.024895 2255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:01.024924 2255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:01.024951 2255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:01.024981 2255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:01.029999 2255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:01.030814 2255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:01.031672 2255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:01.038910 2261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.038985 2260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.038985 2263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.039112 2255 server_base.cc:1034] running on GCE node
I20260927 18:45:01.039664 2255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:01.040606 2255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:01.041754 2255 hybrid_clock.cc:648] HybridClock initialized: now 1790534701041732 us; error 32 us; skew 500 ppm
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:01.044055 2255 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:45:01.044801 2255 webserver.cc:460] Webserver started at http://127.18.235.190:46549/ using document root <none> and password file <none>
I20260927 18:45:01.045301 2255 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:01.045365 2255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:01.045547 2255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:01.047295 2255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3f1c306dee292ea1dbcca79f33538e97"
server_key_iv: "dbb79b2cb835a7914e3f073c1c284e8f"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.047796 2255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3f1c306dee292ea1dbcca79f33538e97"
server_key_iv: "dbb79b2cb835a7914e3f073c1c284e8f"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.051165 2255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:45:01.053725 2270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.054801 2255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:01.054917 2255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3f1c306dee292ea1dbcca79f33538e97"
server_key_iv: "dbb79b2cb835a7914e3f073c1c284e8f"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.055001 2255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:01.070312 2255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:01.072539 2255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:01.072683 2255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:01.078632 2255 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:44351
I20260927 18:45:01.078857 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:44351 every 8 connection(s)
I20260927 18:45:01.079434 2255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:01.081825 2323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:01.082470 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2255
I20260927 18:45:01.082573 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:45:01.082818 19374 external_mini_cluster.cc:1428] Setting key 15361a47c403048bf1e68db51979a4bd
I20260927 18:45:01.087246 2323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap starting.
I20260927 18:45:01.089254 2323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:01.089915 2323 log.cc:824] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:01.091634 2323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: No bootstrap required, opened a new log
I20260927 18:45:01.094959 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.083923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.18.235.190:44351)
0927 18:45:01.084282 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.084295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.084328 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:45:01.084939 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.084955 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.085288 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.085541 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.086353 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.087264 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.088084 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.088523 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.088657 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.091013 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.091018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.091030 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.091054 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.092561 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.092936 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.092941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.092945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.093044 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.093310 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.093313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.093315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.093645 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.093824 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.094069 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.094316 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260927 18:45:01.094913 2323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:01.095100 2323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:01.095175 2323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f93e5c8a66548b98d8ad3fe24e9ca2d, State: Initialized, Role: FOLLOWER
I20260927 18:45:01.095556 2323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:01.095667 2323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:01.095723 2323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:01.095813 2323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:01.096561 2323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:01.096876 2323 leader_election.cc:302] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f93e5c8a66548b98d8ad3fe24e9ca2d; no voters:
I20260927 18:45:01.097180 2323 leader_election.cc:288] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:01.097277 2328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:01.097492 2328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 LEADER]: Becoming Leader. State: Replica: 8f93e5c8a66548b98d8ad3fe24e9ca2d, State: Running, Role: LEADER
I20260927 18:45:01.097801 2328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:01.098214 2323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:01.099208 2329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } } }
I20260927 18:45:01.099327 2329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:01.099684 2337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:01.099668 2330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f93e5c8a66548b98d8ad3fe24e9ca2d. Latest consensus state: current_term: 1 leader_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } } }
I20260927 18:45:01.100636 2330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:01.102524 2337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:01.107136 2337 catalog_manager.cc:1347] Generated new cluster ID: f65414235b84493e83280354f67412df
I20260927 18:45:01.107193 2337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:01.120138 2337 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:01.121228 2337 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:01.129081 2337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Generated new TSK 0
I20260927 18:45:01.129740 2337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:45:01.153352 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44351
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:01.255419 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:01.255623 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:01.255661 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:01.259603 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:01.259649 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:01.259702 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:45:01.264467 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:01.265255 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:01.266083 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:01.274317 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.274384 2349 server_base.cc:1034] running on GCE node
W20260927 18:45:01.274384 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.274507 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.274892 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:01.275418 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:01.276561 2349 hybrid_clock.cc:648] HybridClock initialized: now 1790534701276540 us; error 36 us; skew 500 ppm
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:01.278954 2349 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:45:01.279673 2349 webserver.cc:460] Webserver started at http://127.18.235.129:45241/ using document root <none> and password file <none>
I20260927 18:45:01.280139 2349 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:01.280200 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:01.280365 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:01.282176 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "bbf3cda847534696ae8f14cbb07fd4d8"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3487f89bffc22fd7d1bbd4935cbf02df"
server_key_iv: "057c05f91c6c32137a701c17bbbaa6b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.282667 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "bbf3cda847534696ae8f14cbb07fd4d8"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3487f89bffc22fd7d1bbd4935cbf02df"
server_key_iv: "057c05f91c6c32137a701c17bbbaa6b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.285912 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:45:01.287984 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.289026 2349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:01.289139 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "bbf3cda847534696ae8f14cbb07fd4d8"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3487f89bffc22fd7d1bbd4935cbf02df"
server_key_iv: "057c05f91c6c32137a701c17bbbaa6b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.289233 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:01.319857 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:01.322252 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:01.322408 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:01.322917 2349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:01.323642 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:01.323685 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.323734 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:01.323762 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.331672 2349 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:43461
I20260927 18:45:01.331701 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:43461 every 8 connection(s)
I20260927 18:45:01.332423 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:01.338006 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2349
I20260927 18:45:01.338111 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:01.338408 19374 external_mini_cluster.cc:1428] Setting key 1eadd2b1d5e805fdfb91feb9769528f5
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:01.346024 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.333947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:40981 (local address 127.18.235.190:44351)
0927 18:45:01.334074 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.334079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.334842 (+ 763us) server_negotiation.cc:407] Connection header received
0927 18:45:01.336309 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.336313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.336365 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.336464 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.337518 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.338064 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.339457 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.339775 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.339871 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.342607 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.342611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.342614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.342644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.343981 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.344532 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.344534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.344536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.344613 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.344957 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.344959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.344960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.345181 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.345263 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.345802 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.345903 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:45:01.347019 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.334289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:40981)
0927 18:45:01.334689 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.334726 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.336102 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.336592 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.336598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.336913 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:01.337397 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.337411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.338187 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.338191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.339340 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.339349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.339940 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.339945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.340132 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.340721 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:01.340739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:01.342446 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:01.344094 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.344101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.344115 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.344441 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.344696 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.344699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.344701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.344874 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.345279 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:01.345286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:01.345563 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.345777 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.346082 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:45:01.348182 2478 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44351
I20260927 18:45:01.348380 2478 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:01.348852 2478 heartbeater.cc:511] Master 127.18.235.190:44351 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:45:01.350245 2287 ts_manager.cc:194] Registered new tserver with Master: bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461)
I20260927 18:45:01.351320 2287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:40981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:45:01.363586 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44351
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:01.464958 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:01.465200 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:01.465241 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:01.468997 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:01.469038 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:01.469089 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:45:01.473937 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:01.474725 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:01.475539 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:01.483839 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.483866 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.483866 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.483992 2483 server_base.cc:1034] running on GCE node
I20260927 18:45:01.484440 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:01.484997 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:01.486137 2483 hybrid_clock.cc:648] HybridClock initialized: now 1790534701486143 us; error 78 us; skew 500 ppm
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:01.488550 2483 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:45:01.489348 2483 webserver.cc:460] Webserver started at http://127.18.235.130:41095/ using document root <none> and password file <none>
I20260927 18:45:01.489825 2483 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:01.489886 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:01.490063 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:01.491897 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "c65ed372d68e4503b01b1d7d70c8435f"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "10c75661e894666f557a6f53001319f7"
server_key_iv: "8c24f825ac8324e2d3e8ab13c3952629"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.492393 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "c65ed372d68e4503b01b1d7d70c8435f"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "10c75661e894666f557a6f53001319f7"
server_key_iv: "8c24f825ac8324e2d3e8ab13c3952629"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.495676 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:45:01.497835 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.498950 2483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:01.499045 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "c65ed372d68e4503b01b1d7d70c8435f"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "10c75661e894666f557a6f53001319f7"
server_key_iv: "8c24f825ac8324e2d3e8ab13c3952629"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.499130 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:01.530597 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:01.533402 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:01.533560 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:01.534068 2483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:01.534807 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:01.534853 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.534901 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:01.534927 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.542798 2483 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43025
I20260927 18:45:01.542850 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43025 every 8 connection(s)
I20260927 18:45:01.543596 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:01.548319 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2483
I20260927 18:45:01.548418 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:01.548660 19374 external_mini_cluster.cc:1428] Setting key 3aed7c4bc2be4c457f5045792a3933dd
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:01.556141 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.545135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:53181 (local address 127.18.235.190:44351)
0927 18:45:01.545252 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.545255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.546000 (+ 745us) server_negotiation.cc:407] Connection header received
0927 18:45:01.547191 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.547194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.547239 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.547309 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.548365 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.548970 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.550218 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.550438 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.550523 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.552868 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.552871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.552873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.552897 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.554215 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.554757 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.554760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.554761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.554833 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.555179 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.555181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.555182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.555330 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.555396 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.555931 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.556030 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:45:01.557155 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.545463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:53181)
0927 18:45:01.545856 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.545896 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.547014 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.547445 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.547451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.547760 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:01.548218 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.548232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.549095 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.549098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.550096 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.550106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.550563 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.550567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.550716 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.551199 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:01.551219 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:01.552713 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:01.554327 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.554334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.554348 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.554670 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.554911 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.554914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.554916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.555084 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.555416 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:01.555422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:01.555698 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.555901 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.556207 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:45:01.558142 2612 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44351
I20260927 18:45:01.558320 2612 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:01.558706 2612 heartbeater.cc:511] Master 127.18.235.190:44351 requested a full tablet report, sending...
I20260927 18:45:01.559645 2287 ts_manager.cc:194] Registered new tserver with Master: c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:45:01.560273 2287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:53181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:45:01.572934 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44351
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:01.674669 2617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:01.674872 2617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:01.674909 2617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:01.678773 2617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:01.678818 2617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:01.678880 2617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:45:01.683892 2617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:01.684746 2617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:01.685660 2617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:01.694072 2623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.694124 2625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.694097 2622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.694482 2617 server_base.cc:1034] running on GCE node
I20260927 18:45:01.694808 2617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:01.695350 2617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:01.696497 2617 hybrid_clock.cc:648] HybridClock initialized: now 1790534701696482 us; error 31 us; skew 500 ppm
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:01.698905 2617 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:45:01.699668 2617 webserver.cc:460] Webserver started at http://127.18.235.131:39899/ using document root <none> and password file <none>
I20260927 18:45:01.700165 2617 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:01.700227 2617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:01.700399 2617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:01.702239 2617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "5cb20d4056064b6c8b26b5500e911c03"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "0eeb68b89f78ed7f62f02011e51135dc"
server_key_iv: "79c5e1384ee9a189e9cf2e264768fa2f"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.702735 2617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "5cb20d4056064b6c8b26b5500e911c03"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "0eeb68b89f78ed7f62f02011e51135dc"
server_key_iv: "79c5e1384ee9a189e9cf2e264768fa2f"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.705957 2617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:01.707993 2632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.709036 2617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:01.709159 2617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "5cb20d4056064b6c8b26b5500e911c03"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "0eeb68b89f78ed7f62f02011e51135dc"
server_key_iv: "79c5e1384ee9a189e9cf2e264768fa2f"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.709244 2617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:01.736114 2617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:01.739208 2617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:01.739382 2617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:01.739924 2617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:01.740705 2617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:01.740756 2617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.740808 2617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:01.740841 2617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.749493 2617 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:46847
I20260927 18:45:01.749516 2745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:46847 every 8 connection(s)
I20260927 18:45:01.750283 2617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:01.757517 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2617
I20260927 18:45:01.757611 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:01.757898 19374 external_mini_cluster.cc:1428] Setting key 24c14292b552c75548da0a3bcf3b1ff6
Sep 27 18:45:01 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:01.762450 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.751851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:60271 (local address 127.18.235.190:44351)
0927 18:45:01.751988 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.751992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.752762 (+ 770us) server_negotiation.cc:407] Connection header received
0927 18:45:01.754034 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.754037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.754082 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.754148 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.755129 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.755632 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.756696 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.756904 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.756975 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.759510 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.759513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.759515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.759538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.760722 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.761232 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.761235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.761236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.761316 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.761588 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.761590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.761591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.761740 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.761806 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.762244 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.762338 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:45:01.763151 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.752188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:60271)
0927 18:45:01.752622 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.752661 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.753857 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.754274 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.754280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.754588 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:01.755020 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.755032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.755736 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.755738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.756602 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.756611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.757008 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.757010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.757182 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.757710 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:01.757729 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:01.759360 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:01.760823 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.760827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.760837 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.761136 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.761386 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.761388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.761389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.761519 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.761849 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:01.761854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:01.762055 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.762205 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.762424 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260927 18:45:01.764119 2746 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44351
I20260927 18:45:01.764333 2746 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:01.764740 2746 heartbeater.cc:511] Master 127.18.235.190:44351 requested a full tablet report, sending...
I20260927 18:45:01.765661 2287 ts_manager.cc:194] Registered new tserver with Master: 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847)
I20260927 18:45:01.766191 2287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:60271
I20260927 18:45:01.771296 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:45:01.779259 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.772936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.18.235.190:44351)
0927 18:45:01.773035 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.773038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.773227 (+ 189us) server_negotiation.cc:407] Connection header received
0927 18:45:01.773341 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.773344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.773383 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.773444 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.774059 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.774549 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.775338 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.775552 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.775631 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.776766 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.776769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.776772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.776795 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.777824 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.778180 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.778183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.778185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.778253 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.778492 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.778495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.778497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.778687 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.778795 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.779024 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.779123 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:45:01.781585 2287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:01.784391 2287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:01.797683 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.792088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43025 (local address 127.0.0.1:55712)
0927 18:45:01.792697 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.792709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.792786 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.793850 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.793857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.793881 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:01.794024 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.794033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.795088 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.795092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.796250 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.796261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.797294 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.797298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.797405 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.797437 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.797503 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.797580 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":87,"threads_started":1}
I20260927 18:45:01.798293 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.792577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.18.235.130:43025)
0927 18:45:01.793306 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.793318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.793340 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:45:01.793411 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.793416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.793674 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.793824 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:01.794153 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.794955 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.796365 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.797162 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.797868 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.797992 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.798038 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.798128 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":118,"threads_started":1}
I20260927 18:45:01.798386 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.792753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43461 (local address 127.0.0.1:58380)
0927 18:45:01.793039 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.793051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.793162 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.793995 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.793998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.794024 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:01.794167 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.794174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.795600 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.795604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.796821 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.796830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.797943 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.797948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.798039 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.798066 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.798130 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.798201 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":211,"thread_start_us":98,"threads_started":1}
I20260927 18:45:01.798671 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.791919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46847 (local address 127.0.0.1:52740)
0927 18:45:01.792454 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.792502 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.792625 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.794023 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.794026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.794050 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:01.794171 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.794176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.795630 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.795634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.796780 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.796791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.798086 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.798090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.798224 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.798246 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.798314 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.798537 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":48,"thread_start_us":74,"threads_started":1}
I20260927 18:45:01.799151 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.792903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.18.235.129:43461)
0927 18:45:01.793442 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.793455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.793473 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:45:01.793538 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.793542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.793690 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.793823 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:01.794314 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.795459 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.796958 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.797797 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.798689 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.798817 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.798865 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.798952 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":115,"threads_started":1}
I20260927 18:45:01.799415 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.792039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.18.235.131:46847)
0927 18:45:01.793036 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.793046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.793071 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:45:01.793330 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.793337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.793737 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.793866 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:01.794322 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.795470 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.796904 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.797963 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.798077 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.798256 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.798359 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.799271 (+ 912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":119,"threads_started":1}
I20260927 18:45:01.801178 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 12290bfaf3f747da8c687784c4ecde44 (DEFAULT_TABLE table=test-table [id=630ca43993f84b8d8ecc8b00883fc182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:01.801671 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 12290bfaf3f747da8c687784c4ecde44 (DEFAULT_TABLE table=test-table [id=630ca43993f84b8d8ecc8b00883fc182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:01.802129 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12290bfaf3f747da8c687784c4ecde44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:01.802734 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12290bfaf3f747da8c687784c4ecde44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:01.803030 2680 tablet_service.cc:1467] Processing CreateTablet for tablet 12290bfaf3f747da8c687784c4ecde44 (DEFAULT_TABLE table=test-table [id=630ca43993f84b8d8ecc8b00883fc182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:01.804004 2680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12290bfaf3f747da8c687784c4ecde44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:01.809446 2763 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Bootstrap starting.
I20260927 18:45:01.810156 2764 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap starting.
I20260927 18:45:01.810568 2765 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Bootstrap starting.
I20260927 18:45:01.812103 2763 tablet_bootstrap.cc:654] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:01.812227 2764 tablet_bootstrap.cc:654] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:01.812436 2765 tablet_bootstrap.cc:654] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:01.812970 2763 log.cc:824] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:01.813450 2764 log.cc:824] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:01.813655 2765 log.cc:824] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:01.814999 2763 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: No bootstrap required, opened a new log
I20260927 18:45:01.815200 2765 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: No bootstrap required, opened a new log
I20260927 18:45:01.815205 2763 ts_tablet_manager.cc:1397] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:45:01.815299 2764 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: No bootstrap required, opened a new log
I20260927 18:45:01.815362 2765 ts_tablet_manager.cc:1397] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:45:01.815469 2764 ts_tablet_manager.cc:1397] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:45:01.818050 2765 raft_consensus.cc:348] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:01.818255 2765 raft_consensus.cc:374] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:01.818324 2765 raft_consensus.cc:729] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb20d4056064b6c8b26b5500e911c03, State: Initialized, Role: FOLLOWER
I20260927 18:45:01.818738 2765 consensus_queue.cc:260] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:01.819054 2764 raft_consensus.cc:348] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:01.819054 2763 raft_consensus.cc:348] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:01.819283 2746 heartbeater.cc:503] Master 127.18.235.190:44351 was elected leader, sending a full tablet report...
I20260927 18:45:01.819303 2763 raft_consensus.cc:374] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:01.819356 2763 raft_consensus.cc:729] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65ed372d68e4503b01b1d7d70c8435f, State: Initialized, Role: FOLLOWER
I20260927 18:45:01.819451 2764 raft_consensus.cc:374] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:01.819500 2764 raft_consensus.cc:729] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf3cda847534696ae8f14cbb07fd4d8, State: Initialized, Role: FOLLOWER
I20260927 18:45:01.819685 2765 ts_tablet_manager.cc:1428] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 18:45:01.819919 2764 consensus_queue.cc:260] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:01.820159 2763 consensus_queue.cc:260] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:01.820633 2478 heartbeater.cc:503] Master 127.18.235.190:44351 was elected leader, sending a full tablet report...
I20260927 18:45:01.820832 2764 ts_tablet_manager.cc:1428] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:01.821025 2763 ts_tablet_manager.cc:1428] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:45:01.821339 2612 heartbeater.cc:503] Master 127.18.235.190:44351 was elected leader, sending a full tablet report...
W20260927 18:45:01.833338 2479 tablet.cc:2369] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:45:02.001180 2747 tablet.cc:2369] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:45:02.044552 2613 tablet.cc:2369] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:02.115685 2770 raft_consensus.cc:482] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:02.115877 2770 raft_consensus.cc:504] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:02.116971 2770 leader_election.cc:288] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
I20260927 18:45:02.120532 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.117312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:41917 (local address 127.18.235.130:43025)
0927 18:45:02.117430 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.117434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.117611 (+ 177us) server_negotiation.cc:407] Connection header received
0927 18:45:02.117725 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.117727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.117769 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.117839 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:02.118079 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.118553 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.119656 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.120020 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.120113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.120174 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.120342 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.120407 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:45:02.120554 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.117292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:48721 (local address 127.18.235.131:46847)
0927 18:45:02.117511 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.117515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.117534 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:45:02.117674 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.117677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.117725 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.117778 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:02.118053 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.118538 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.119653 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.120039 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.120143 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.120193 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.120381 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.120454 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 18:45:02.120560 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.117187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46847 (local address 127.18.235.129:48721)
0927 18:45:02.117462 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:02.117475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:02.117567 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:02.117808 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:02.117812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:02.117828 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:02.117949 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.117956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.118641 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.118659 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.119535 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.119545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.120150 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.120155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.120267 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.120284 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:45:02.120358 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:45:02.120420 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":35,"thread_start_us":104,"threads_started":1}
I20260927 18:45:02.120568 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.117201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43025 (local address 127.18.235.129:41917)
0927 18:45:02.117515 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:02.117528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:02.117629 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:02.117868 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:02.117871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:02.117887 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:02.117975 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.117979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.118655 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.118657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.119537 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.119546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.120130 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.120133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.120232 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.120254 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:45:02.120318 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:45:02.120384 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":61,"thread_start_us":105,"threads_started":1}
I20260927 18:45:02.121220 2700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12290bfaf3f747da8c687784c4ecde44" candidate_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb20d4056064b6c8b26b5500e911c03" is_pre_election: true
I20260927 18:45:02.121325 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12290bfaf3f747da8c687784c4ecde44" candidate_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65ed372d68e4503b01b1d7d70c8435f" is_pre_election: true
I20260927 18:45:02.121541 2566 raft_consensus.cc:2454] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf3cda847534696ae8f14cbb07fd4d8 in term 0.
I20260927 18:45:02.121533 2700 raft_consensus.cc:2454] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf3cda847534696ae8f14cbb07fd4d8 in term 0.
I20260927 18:45:02.121989 2366 leader_election.cc:302] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8, c65ed372d68e4503b01b1d7d70c8435f; no voters:
I20260927 18:45:02.122232 2770 raft_consensus.cc:2781] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:02.122306 2770 raft_consensus.cc:482] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:02.122349 2770 raft_consensus.cc:3037] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:02.123273 2770 raft_consensus.cc:504] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:02.123564 2770 leader_election.cc:288] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 1 election: Requested vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
I20260927 18:45:02.123840 2700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12290bfaf3f747da8c687784c4ecde44" candidate_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb20d4056064b6c8b26b5500e911c03"
I20260927 18:45:02.123881 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12290bfaf3f747da8c687784c4ecde44" candidate_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65ed372d68e4503b01b1d7d70c8435f"
I20260927 18:45:02.123941 2700 raft_consensus.cc:3037] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:02.123976 2566 raft_consensus.cc:3037] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:02.124878 2700 raft_consensus.cc:2454] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf3cda847534696ae8f14cbb07fd4d8 in term 1.
I20260927 18:45:02.124919 2566 raft_consensus.cc:2454] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf3cda847534696ae8f14cbb07fd4d8 in term 1.
I20260927 18:45:02.125166 2367 leader_election.cc:302] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb20d4056064b6c8b26b5500e911c03, bbf3cda847534696ae8f14cbb07fd4d8; no voters:
I20260927 18:45:02.125308 2770 raft_consensus.cc:2781] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:02.125486 2770 raft_consensus.cc:686] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 LEADER]: Becoming Leader. State: Replica: bbf3cda847534696ae8f14cbb07fd4d8, State: Running, Role: LEADER
I20260927 18:45:02.125788 2770 consensus_queue.cc:237] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:02.128571 2287 catalog_manager.cc:5495] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129). New cstate: current_term: 1 leader_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } health_report { overall_health: HEALTHY } } }
I20260927 18:45:02.153805 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.150626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.18.235.129:43461)
0927 18:45:02.150733 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.150737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.150753 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:45:02.150910 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.150912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.150957 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.151036 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:02.151261 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.151723 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.152522 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.152731 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.152812 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.153076 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:02.153468 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:02.153569 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.153640 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.153689 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:45:02.158957 2566 raft_consensus.cc:1261] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Refusing update from remote peer bbf3cda847534696ae8f14cbb07fd4d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:02.158949 2700 raft_consensus.cc:1261] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Refusing update from remote peer bbf3cda847534696ae8f14cbb07fd4d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:02.159586 2774 consensus_queue.cc:1035] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:02.159742 2770 consensus_queue.cc:1035] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:02.169202 2779 mvcc.cc:205] Tried to move back new op lower bound from 7334030140036763648 to 7334030139909586944. Current Snapshot: MvccSnapshot[applied={T|T < 7334030140036763648 or (T in {7334030140036763648})}]
I20260927 18:45:02.173647 2287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:02.173772 2287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:02.175729 2287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Sending DeleteTablet for 3 replicas of tablet 12290bfaf3f747da8c687784c4ecde44
I20260927 18:45:02.176417 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2349
I20260927 18:45:02.176486 2680 tablet_service.cc:1514] Processing DeleteTablet for tablet 12290bfaf3f747da8c687784c4ecde44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:02 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:52740
I20260927 18:45:02.176584 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 12290bfaf3f747da8c687784c4ecde44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:02 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:55712
I20260927 18:45:02.177219 2788 tablet_replica.cc:330] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: stopping tablet replica
I20260927 18:45:02.177281 2787 tablet_replica.cc:330] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: stopping tablet replica
I20260927 18:45:02.177479 2788 raft_consensus.cc:2229] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:02.177477 2787 raft_consensus.cc:2229] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:02.177732 2788 raft_consensus.cc:2258] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:02.177755 2787 raft_consensus.cc:2258] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:02.179065 2788 ts_tablet_manager.cc:1905] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:02.179065 2787 ts_tablet_manager.cc:1905] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:02.181571 2788 ts_tablet_manager.cc:1918] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:02.181694 2788 log.cc:1196] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/12290bfaf3f747da8c687784c4ecde44
I20260927 18:45:02.182001 2788 ts_tablet_manager.cc:1939] T 12290bfaf3f747da8c687784c4ecde44 P c65ed372d68e4503b01b1d7d70c8435f: Deleting consensus metadata
W20260927 18:45:02.182006 2273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:43461 (error 108)
I20260927 18:45:02.182294 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2483
I20260927 18:45:02.182394 2787 ts_tablet_manager.cc:1918] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:02.182927 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.182622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43461 (local address 127.0.0.1:58394)
0927 18:45:02.182810 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:02.182865 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 18:45:02.183082 2787 log.cc:1196] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/12290bfaf3f747da8c687784c4ecde44
W20260927 18:45:02.183117 2273 catalog_manager.cc:4584] TS bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12290bfaf3f747da8c687784c4ecde44: Network error: Client connection negotiation failed: client connection to 127.18.235.129:43461: connect: Connection refused (error 111)
I20260927 18:45:02.183386 2787 ts_tablet_manager.cc:1939] T 12290bfaf3f747da8c687784c4ecde44 P 5cb20d4056064b6c8b26b5500e911c03: Deleting consensus metadata
I20260927 18:45:02.184269 2273 catalog_manager.cc:4841] TS 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): tablet 12290bfaf3f747da8c687784c4ecde44 (table test-table [id=630ca43993f84b8d8ecc8b00883fc182]) successfully deleted
I20260927 18:45:02.187691 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2617
I20260927 18:45:02.188298 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.187800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43025 (local address 127.0.0.1:55716)
0927 18:45:02.188195 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:02.188242 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.130:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260927 18:45:02.192268 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2255
I20260927 18:45:22.197495 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44351
--webserver_interface=127.18.235.190
--webserver_port=46549
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:22.300339 2790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:22.300551 2790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:22.300586 2790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:22.304433 2790 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:45:22.304473 2790 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:45:22.304488 2790 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:45:22.304504 2790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:22.304524 2790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:22.304546 2790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:22.304561 2790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:22.304577 2790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:22.309441 2790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:22.310247 2790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:22.311080 2790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.322348 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.322376 2790 server_base.cc:1034] running on GCE node
W20260927 18:45:22.322346 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.322343 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.323103 2790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.324120 2790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.325265 2790 hybrid_clock.cc:648] HybridClock initialized: now 1790534722325245 us; error 35 us; skew 500 ppm
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.327761 2790 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:45:22.328544 2790 webserver.cc:460] Webserver started at http://127.18.235.190:46549/ using document root <none> and password file <none>
I20260927 18:45:22.329046 2790 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.329133 2790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.332638 2790 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 18:45:22.334573 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.335659 2790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:45:22.335763 2790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3f1c306dee292ea1dbcca79f33538e97"
server_key_iv: "dbb79b2cb835a7914e3f073c1c284e8f"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.336304 2790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.351106 2790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.353850 2790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:22.354014 2790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.360146 2790 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:44351
I20260927 18:45:22.360383 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:44351 every 8 connection(s)
I20260927 18:45:22.361099 2790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:22.362882 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2790
I20260927 18:45:22.363312 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:43461
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:44351
--webserver_port=45241
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:22.365589 2859 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:45:22.367143 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap starting.
I20260927 18:45:22.383097 2859 log.cc:824] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:22.392131 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:22.392498 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap complete.
I20260927 18:45:22.396770 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:22.397192 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f93e5c8a66548b98d8ad3fe24e9ca2d, State: Initialized, Role: FOLLOWER
I20260927 18:45:22.397629 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:22.397732 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:22.397795 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:22.397907 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:45:22.399605 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:22.399899 2859 leader_election.cc:302] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f93e5c8a66548b98d8ad3fe24e9ca2d; no voters:
I20260927 18:45:22.400164 2859 leader_election.cc:288] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:45:22.400246 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:45:22.400485 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 LEADER]: Becoming Leader. State: Replica: 8f93e5c8a66548b98d8ad3fe24e9ca2d, State: Running, Role: LEADER
I20260927 18:45:22.400808 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:22.401404 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:22.401861 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f93e5c8a66548b98d8ad3fe24e9ca2d. Latest consensus state: current_term: 2 leader_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } } }
I20260927 18:45:22.402079 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:22.402205 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } } }
I20260927 18:45:22.402369 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:22.402506 2868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:22.407049 2868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:22.407819 2868 catalog_manager.cc:1259] Loaded cluster ID: f65414235b84493e83280354f67412df
I20260927 18:45:22.407886 2868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:22.410296 2868 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:22.411019 2868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Loaded TSK: 0
I20260927 18:45:22.430886 2868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Generated new TSK 1
I20260927 18:45:22.431114 2868 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:45:22.470345 2861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:22.470551 2861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:22.470605 2861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:22.474537 2861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:22.474582 2861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:22.474654 2861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:45:22.479513 2861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:43461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=45241
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:22.480306 2861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:22.481259 2861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.489379 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.489400 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.489837 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.490001 2861 server_base.cc:1034] running on GCE node
I20260927 18:45:22.490324 2861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.490862 2861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.492027 2861 hybrid_clock.cc:648] HybridClock initialized: now 1790534722492009 us; error 36 us; skew 500 ppm
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.494552 2861 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:45:22.495322 2861 webserver.cc:460] Webserver started at http://127.18.235.129:45241/ using document root <none> and password file <none>
I20260927 18:45:22.495788 2861 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.495862 2861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.499337 2861 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:22.501240 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.502381 2861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:22.502526 2861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "bbf3cda847534696ae8f14cbb07fd4d8"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3487f89bffc22fd7d1bbd4935cbf02df"
server_key_iv: "057c05f91c6c32137a701c17bbbaa6b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.503010 2861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.532411 2861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.535166 2861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:22.535318 2861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.535851 2861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:22.536871 2903 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:45:22.540777 2861 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:45:22.540833 2861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 18:45:22.540877 2861 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:45:22.542434 2861 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:45:22.542490 2861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 18:45:22.542742 2903 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap starting.
I20260927 18:45:22.552347 2861 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:43461
I20260927 18:45:22.552367 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:43461 every 8 connection(s)
I20260927 18:45:22.553485 2861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:22.559206 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 2861
I20260927 18:45:22.559686 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:43025
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:44351
--webserver_port=41095
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:22.562042 2903 log.cc:824] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:22.569303 2903 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:22.569737 2903 tablet_bootstrap.cc:492] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap complete.
I20260927 18:45:22.570943 2903 ts_tablet_manager.cc:1397] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.004s
I20260927 18:45:22.570989 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.555457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:41445)
0927 18:45:22.555940 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.555980 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.556731 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.557842 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.557850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.558241 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.558666 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.558679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.560416 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.560420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.561726 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.561737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.562603 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.562607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.562796 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.563428 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.563446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.565271 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.567681 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.567687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.567698 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.568034 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.568444 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.568447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.568449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.568614 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.569328 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.569334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.569610 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.569821 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.570120 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260927 18:45:22.571010 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.555399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:41445 (local address 127.18.235.190:44351)
0927 18:45:22.555732 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.555757 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.556106 (+ 349us) server_negotiation.cc:407] Connection header received
0927 18:45:22.557143 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.557158 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.557462 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.557761 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.558838 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.560217 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.561875 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.562446 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.562616 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.565470 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.565476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.565492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.565525 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.567554 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.568168 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.568174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.568179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.568343 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.568718 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.568722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.568724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.569192 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.569338 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.569948 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.570234 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
I20260927 18:45:22.573637 3011 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44351
I20260927 18:45:22.573872 3011 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:22.573529 2903 raft_consensus.cc:348] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.574010 2903 raft_consensus.cc:729] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf3cda847534696ae8f14cbb07fd4d8, State: Initialized, Role: FOLLOWER
I20260927 18:45:22.574402 3011 heartbeater.cc:511] Master 127.18.235.190:44351 requested a full tablet report, sending...
I20260927 18:45:22.574467 2903 consensus_queue.cc:260] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.575407 2903 ts_tablet_manager.cc:1428] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 18:45:22.576942 2823 ts_manager.cc:194] Registered new tserver with Master: bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461)
I20260927 18:45:22.580580 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:41445
I20260927 18:45:22.583285 3011 heartbeater.cc:503] Master 127.18.235.190:44351 was elected leader, sending a full tablet report...
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.129@KRBTEST.COM
I20260927 18:45:22.589819 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.579945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43461 (local address 127.0.0.1:59080)
0927 18:45:22.580294 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.580335 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.580452 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.581053 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.581059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.581438 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.581552 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.581559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.582805 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.582809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.583701 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.583712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.584417 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.584421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.584523 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.585021 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.585035 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.586472 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.588336 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.588344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.588348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.588554 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.588823 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.588825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.588827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.588892 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.589355 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.589360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.589452 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.589556 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.589691 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
I20260927 18:45:22.589947 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.580007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.18.235.129:43461)
0927 18:45:22.580657 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.580666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.580685 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:45:22.580748 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.580754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.580903 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.581016 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.581690 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.582679 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.583812 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.584279 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.584396 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.586614 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.586623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.586633 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.586657 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.588205 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.588650 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.588656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.588658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.588742 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.588960 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.588963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.588965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.589249 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.589381 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.589652 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.589802 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":84,"threads_started":1}
I20260927 18:45:22.590783 2945 tablet_service.cc:1514] Processing DeleteTablet for tablet 12290bfaf3f747da8c687784c4ecde44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:02 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:59080
I20260927 18:45:22.591626 3020 tablet_replica.cc:330] stopping tablet replica
I20260927 18:45:22.591871 3020 raft_consensus.cc:2229] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:22.592072 3020 raft_consensus.cc:2258] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:22.592887 3020 ts_tablet_manager.cc:1905] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:22.596061 3020 ts_tablet_manager.cc:1918] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:22.596143 3020 log.cc:1196] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal/wals/12290bfaf3f747da8c687784c4ecde44
I20260927 18:45:22.596345 3020 ts_tablet_manager.cc:1939] T 12290bfaf3f747da8c687784c4ecde44 P bbf3cda847534696ae8f14cbb07fd4d8: Deleting consensus metadata
I20260927 18:45:22.597982 2809 catalog_manager.cc:4841] TS bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461): tablet 12290bfaf3f747da8c687784c4ecde44 (table test-table [id=630ca43993f84b8d8ecc8b00883fc182]) successfully deleted
W20260927 18:45:22.670784 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:22.670987 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:22.671025 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:22.674887 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:22.674933 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:22.674986 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:45:22.679842 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:43025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=41095
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:22.680612 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:22.681483 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.689755 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.689791 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.689908 3015 server_base.cc:1034] running on GCE node
W20260927 18:45:22.689953 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.690358 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.690893 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.692030 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790534722692008 us; error 36 us; skew 500 ppm
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.694425 3015 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:45:22.695139 3015 webserver.cc:460] Webserver started at http://127.18.235.130:41095/ using document root <none> and password file <none>
I20260927 18:45:22.695641 3015 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.695713 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.699095 3015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:22.700820 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.701910 3015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:22.701999 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "c65ed372d68e4503b01b1d7d70c8435f"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "10c75661e894666f557a6f53001319f7"
server_key_iv: "8c24f825ac8324e2d3e8ab13c3952629"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.702525 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.739974 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.742525 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:22.742669 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.743160 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:22.743865 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:22.743906 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.743944 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:22.743973 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.751700 3015 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43025
I20260927 18:45:22.751729 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43025 every 8 connection(s)
I20260927 18:45:22.752530 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:22.756122 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3015
I20260927 18:45:22.756589 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:46847
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:44351
--webserver_port=39899
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:22.765579 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.754027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:34909 (local address 127.18.235.190:44351)
0927 18:45:22.754142 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.754146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.754879 (+ 733us) server_negotiation.cc:407] Connection header received
0927 18:45:22.756121 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.756125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.756183 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.756259 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.757668 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.758274 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.759503 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.759740 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.759827 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.762357 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.762360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.762362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.762385 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.763620 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.764161 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.764163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.764164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.764228 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.764569 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.764571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.764572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.764723 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.764795 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.765356 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.765461 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:45:22.766563 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.754359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:34909)
0927 18:45:22.754746 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.754787 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.755921 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.756457 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.756466 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.756894 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.757499 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.757516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.758402 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.758404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.759374 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.759385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.759878 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.759881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.760069 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.760589 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.760608 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.762189 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.763728 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.763735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.763749 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.764074 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.764306 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.764310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.764312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.764490 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.764813 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.764819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.765097 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.765321 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.765633 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20260927 18:45:22.767601 3149 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44351
I20260927 18:45:22.767791 3149 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:22.768175 3149 heartbeater.cc:511] Master 127.18.235.190:44351 requested a full tablet report, sending...
I20260927 18:45:22.769073 2823 ts_manager.cc:194] Registered new tserver with Master: c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
I20260927 18:45:22.769639 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:34909
W20260927 18:45:22.861445 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:22.861673 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:22.861728 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:22.865604 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:22.865653 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:22.865736 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:45:22.870609 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:46847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=39899
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:22.871446 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:22.872339 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.880963 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.880935 3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.880946 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.881042 3152 server_base.cc:1034] running on GCE node
I20260927 18:45:22.881611 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.882174 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.883329 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790534722883306 us; error 37 us; skew 500 ppm
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.885784 3152 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:45:22.886673 3152 webserver.cc:460] Webserver started at http://127.18.235.131:39899/ using document root <none> and password file <none>
I20260927 18:45:22.887167 3152 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.887248 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.890710 3152 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 18:45:22.892525 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.893747 3152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:22.893847 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "5cb20d4056064b6c8b26b5500e911c03"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "0eeb68b89f78ed7f62f02011e51135dc"
server_key_iv: "79c5e1384ee9a189e9cf2e264768fa2f"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.894307 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.904696 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.907107 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:22.907246 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.907711 3152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:22.908387 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:22.908437 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.908483 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:22.908506 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.916235 3152 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:46847
I20260927 18:45:22.916260 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:46847 every 8 connection(s)
I20260927 18:45:22.917044 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:22.922286 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3152
Sep 27 18:45:22 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:45:22.929347 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.918576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:59225 (local address 127.18.235.190:44351)
0927 18:45:22.918688 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.918691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.919385 (+ 694us) server_negotiation.cc:407] Connection header received
0927 18:45:22.920624 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.920627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.920682 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.920750 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.921814 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.922349 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.923632 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.923839 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.923927 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.926177 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.926180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.926183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.926207 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.927514 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.928040 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.928042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.928044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.928111 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.928437 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.928439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.928440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.928581 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.928646 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.929060 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.929198 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:45:22.930169 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.918892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:59225)
0927 18:45:22.919262 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.919298 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.920437 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.920909 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.920916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.921259 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.921681 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.921694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.922453 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.922457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.923533 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.923541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.923956 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.923960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.924122 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.924579 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.924594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.926025 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.927612 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.927618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.927631 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.927950 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.928191 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.928194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.928196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.928366 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.928654 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.928658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.928847 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.929006 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.929302 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260927 18:45:22.930261 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.922837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.18.235.190:44351)
0927 18:45:22.923073 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.923077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.923141 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:45:22.923230 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.923232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.923272 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.923354 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.923988 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.924486 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.925334 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.925547 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.925626 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.926910 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.926913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.926916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.926949 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.928356 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.928938 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.928942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.928943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.929026 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.929319 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.929322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.929324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.929508 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.929694 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.930010 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.930096 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260927 18:45:22.931317 3281 heartbeater.cc:347] Connected to a master server at 127.18.235.190:44351
I20260927 18:45:22.931528 3281 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:22.932018 3281 heartbeater.cc:511] Master 127.18.235.190:44351 requested a full tablet report, sending...
I20260927 18:45:22.932853 2823 ts_manager.cc:194] Registered new tserver with Master: 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847)
I20260927 18:45:22.933399 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:59225
I20260927 18:45:22.943262 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:45:22.946837 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.943747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.18.235.190:44351)
0927 18:45:22.943859 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.943863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.944018 (+ 155us) server_negotiation.cc:407] Connection header received
0927 18:45:22.944163 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.944166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.944244 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.944321 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:22.944525 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.944995 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.945832 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.946058 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.946141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.946419 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:22.946530 (+ 111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 18:45:22.946733 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54472: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 18:45:22.946897 3285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54472: token expired
W20260927 18:45:22.947172 2749 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.235.190:44351)
I20260927 18:45:22.952783 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.947455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.18.235.190:44351)
0927 18:45:22.947555 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.947558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.947570 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:45:22.947639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.947641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.947669 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.947726 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.948204 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.948639 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.949378 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.949569 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.949631 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.950617 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.950620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.950622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.950640 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.951669 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.951997 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.951999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.952000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.952066 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.952237 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.952239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.952240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.952369 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.952453 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.952587 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.952660 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:45:22.954706 2823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:22.957719 2823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:22.964363 2945 tablet_service.cc:1467] Processing CreateTablet for tablet 750ad6cb5f914983bc962da89b81cb31 (DEFAULT_TABLE table=test-table [id=7957cee3949c4f6988f2d37dff106a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:22.964937 2945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750ad6cb5f914983bc962da89b81cb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:22.967619 2903 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap starting.
I20260927 18:45:22.967805 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.963256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46847 (local address 127.0.0.1:38632)
0927 18:45:22.963369 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.963382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.963455 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.964106 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.964108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.964125 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:22.964199 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.964205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.965687 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.965690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.966696 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.966704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.967421 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.967424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.967515 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.967552 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.967623 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.967693 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":19}
I20260927 18:45:22.967866 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.963386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.18.235.131:46847)
0927 18:45:22.963663 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.963674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.963693 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:45:22.963755 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.963760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.963959 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.964335 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:22.964472 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.965572 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.966803 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.967311 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.967422 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.967537 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.967661 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.967748 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":112,"threads_started":1}
I20260927 18:45:22.968454 2903 tablet_bootstrap.cc:654] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:22.969239 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.963596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43025 (local address 127.0.0.1:54756)
0927 18:45:22.963867 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.963879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.963949 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.964650 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.964653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.964672 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:22.964751 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.964757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.966296 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.966299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.967383 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.967393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.968861 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.968865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.968951 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.968973 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.969044 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.969124 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260927 18:45:22.969333 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.963846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.18.235.130:43025)
0927 18:45:22.964183 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.964194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.964216 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:45:22.964287 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.964292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.964482 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.964658 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:22.964849 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.966183 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.967521 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.968719 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.968847 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.968971 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.969069 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.969188 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260927 18:45:22.970139 2903 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: No bootstrap required, opened a new log
I20260927 18:45:22.970227 2903 ts_tablet_manager.cc:1397] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:22.970598 2903 raft_consensus.cc:348] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.970688 2903 raft_consensus.cc:374] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:22.970724 2903 raft_consensus.cc:729] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf3cda847534696ae8f14cbb07fd4d8, State: Initialized, Role: FOLLOWER
I20260927 18:45:22.970836 2903 consensus_queue.cc:260] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.970604 3215 tablet_service.cc:1467] Processing CreateTablet for tablet 750ad6cb5f914983bc962da89b81cb31 (DEFAULT_TABLE table=test-table [id=7957cee3949c4f6988f2d37dff106a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:22.971069 2903 ts_tablet_manager.cc:1428] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:45:22.971613 3215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750ad6cb5f914983bc962da89b81cb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:22.972184 3083 tablet_service.cc:1467] Processing CreateTablet for tablet 750ad6cb5f914983bc962da89b81cb31 (DEFAULT_TABLE table=test-table [id=7957cee3949c4f6988f2d37dff106a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:22.973143 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750ad6cb5f914983bc962da89b81cb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:22.979206 3294 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Bootstrap starting.
I20260927 18:45:22.980159 3295 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Bootstrap starting.
I20260927 18:45:22.981520 3294 tablet_bootstrap.cc:654] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:22.982263 3294 log.cc:824] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:22.982286 3295 tablet_bootstrap.cc:654] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:22.982882 3295 log.cc:824] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:22.983920 3294 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: No bootstrap required, opened a new log
I20260927 18:45:22.984081 3294 ts_tablet_manager.cc:1397] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:45:22.984411 3295 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: No bootstrap required, opened a new log
I20260927 18:45:22.984562 3295 ts_tablet_manager.cc:1397] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:45:22.986975 3294 raft_consensus.cc:348] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.987226 3294 raft_consensus.cc:374] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:22.987298 3294 raft_consensus.cc:729] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb20d4056064b6c8b26b5500e911c03, State: Initialized, Role: FOLLOWER
I20260927 18:45:22.987149 3295 raft_consensus.cc:348] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.987377 3295 raft_consensus.cc:374] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:22.987439 3295 raft_consensus.cc:729] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65ed372d68e4503b01b1d7d70c8435f, State: Initialized, Role: FOLLOWER
I20260927 18:45:22.987752 3294 consensus_queue.cc:260] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.987777 3295 consensus_queue.cc:260] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:22.988322 3281 heartbeater.cc:503] Master 127.18.235.190:44351 was elected leader, sending a full tablet report...
I20260927 18:45:22.988510 3149 heartbeater.cc:503] Master 127.18.235.190:44351 was elected leader, sending a full tablet report...
I20260927 18:45:22.988586 3294 ts_tablet_manager.cc:1428] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 18:45:22.989296 3295 ts_tablet_manager.cc:1428] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 18:45:23.003257 3150 tablet.cc:2369] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:45:23.054591 3012 tablet.cc:2369] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:23.141355 3299 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:23.141531 3299 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:23.142553 3299 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461)
I20260927 18:45:23.146750 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.142759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46847 (local address 127.18.235.130:34443)
0927 18:45:23.142886 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.142913 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.143005 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.143273 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.143276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.143297 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:23.143430 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.143437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.144375 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.144378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.145576 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.145587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.146373 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.146376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.146459 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.146481 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.146552 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.146623 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 18:45:23.146759 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.142854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:34443 (local address 127.18.235.131:46847)
0927 18:45:23.143001 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.143006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.143024 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:45:23.143082 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.143085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.143133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.143208 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:23.143534 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.144276 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.145684 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.146274 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.146391 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.146440 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.146576 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.146641 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:45:23.146872 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.143111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:51003 (local address 127.18.235.129:43461)
0927 18:45:23.143369 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.143373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.143385 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:45:23.143501 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.143504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.143568 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.143670 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:23.143866 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.144515 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.145807 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.146417 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.146523 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.146662 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.146698 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.146759 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":60,"threads_started":1}
I20260927 18:45:23.147063 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.143003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43461 (local address 127.18.235.130:51003)
0927 18:45:23.143316 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.143329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.143400 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.143676 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.143678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.143694 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:23.143779 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.143783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.144598 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.144601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.145719 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.145726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.146519 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.146523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.146614 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.146632 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.146682 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.146731 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260927 18:45:23.147588 2965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750ad6cb5f914983bc962da89b81cb31" candidate_uuid: "c65ed372d68e4503b01b1d7d70c8435f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" is_pre_election: true
I20260927 18:45:23.147601 3235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750ad6cb5f914983bc962da89b81cb31" candidate_uuid: "c65ed372d68e4503b01b1d7d70c8435f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb20d4056064b6c8b26b5500e911c03" is_pre_election: true
I20260927 18:45:23.147861 2965 raft_consensus.cc:2454] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c65ed372d68e4503b01b1d7d70c8435f in term 0.
I20260927 18:45:23.147890 3235 raft_consensus.cc:2454] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c65ed372d68e4503b01b1d7d70c8435f in term 0.
I20260927 18:45:23.148247 3038 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8, c65ed372d68e4503b01b1d7d70c8435f; no voters:
I20260927 18:45:23.148473 3299 raft_consensus.cc:2781] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:23.148555 3299 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:23.148605 3299 raft_consensus.cc:3037] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:23.149585 3299 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:23.149852 3299 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [CANDIDATE]: Term 1 election: Requested vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461)
I20260927 18:45:23.150206 3235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750ad6cb5f914983bc962da89b81cb31" candidate_uuid: "c65ed372d68e4503b01b1d7d70c8435f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb20d4056064b6c8b26b5500e911c03"
I20260927 18:45:23.150207 2965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750ad6cb5f914983bc962da89b81cb31" candidate_uuid: "c65ed372d68e4503b01b1d7d70c8435f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf3cda847534696ae8f14cbb07fd4d8"
I20260927 18:45:23.150300 3235 raft_consensus.cc:3037] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:23.150307 2965 raft_consensus.cc:3037] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:23.151214 3235 raft_consensus.cc:2454] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c65ed372d68e4503b01b1d7d70c8435f in term 1.
I20260927 18:45:23.151255 2965 raft_consensus.cc:2454] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c65ed372d68e4503b01b1d7d70c8435f in term 1.
I20260927 18:45:23.151465 3038 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb20d4056064b6c8b26b5500e911c03, c65ed372d68e4503b01b1d7d70c8435f; no voters:
I20260927 18:45:23.151607 3299 raft_consensus.cc:2781] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:23.151780 3299 raft_consensus.cc:686] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 1 LEADER]: Becoming Leader. State: Replica: c65ed372d68e4503b01b1d7d70c8435f, State: Running, Role: LEADER
I20260927 18:45:23.152096 3299 consensus_queue.cc:237] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:23.154855 2823 catalog_manager.cc:5495] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f reported cstate change: term changed from 0 to 1, leader changed from <none> to c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130). New cstate: current_term: 1 leader_uuid: "c65ed372d68e4503b01b1d7d70c8435f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } health_report { overall_health: UNKNOWN } } }
W20260927 18:45:23.167829 3282 tablet.cc:2369] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:23.199872 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.196866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.18.235.130:43025)
0927 18:45:23.196968 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.196972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.196985 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:45:23.197144 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.197148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.197205 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.197283 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:23.197489 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.197962 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.198758 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.198973 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.199048 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.199237 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:23.199610 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:23.199691 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.199718 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.199762 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:45:23.205358 2965 raft_consensus.cc:1261] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Refusing update from remote peer c65ed372d68e4503b01b1d7d70c8435f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:23.205361 3235 raft_consensus.cc:1261] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Refusing update from remote peer c65ed372d68e4503b01b1d7d70c8435f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:23.205976 3299 consensus_queue.cc:1035] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:23.206175 3302 consensus_queue.cc:1035] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:23.211057 3306 mvcc.cc:205] Tried to move back new op lower bound from 7334030226242482176 to 7334030226033266688. Current Snapshot: MvccSnapshot[applied={T|T < 7334030226242482176}]
I20260927 18:45:23.219002 2821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:23.219130 2821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:23.221163 2821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Sending DeleteTablet for 3 replicas of tablet 750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:23.221777 2945 tablet_service.cc:1514] Processing DeleteTablet for tablet 750ad6cb5f914983bc962da89b81cb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:59080
I20260927 18:45:23.221794 3083 tablet_service.cc:1514] Processing DeleteTablet for tablet 750ad6cb5f914983bc962da89b81cb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:54756
I20260927 18:45:23.221874 3215 tablet_service.cc:1514] Processing DeleteTablet for tablet 750ad6cb5f914983bc962da89b81cb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:23 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:38632
I20260927 18:45:23.222173 3316 tablet_replica.cc:330] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: stopping tablet replica
I20260927 18:45:23.222283 3317 tablet_replica.cc:330] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: stopping tablet replica
I20260927 18:45:23.222443 3316 raft_consensus.cc:2229] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:23.222537 3317 raft_consensus.cc:2229] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:23.222594 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2861
I20260927 18:45:23.222723 3316 raft_consensus.cc:2258] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:23.222783 3317 raft_consensus.cc:2258] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:23.224179 3316 ts_tablet_manager.cc:1905] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:23.226287 3317 ts_tablet_manager.cc:1905] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:23.226768 3316 ts_tablet_manager.cc:1918] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:23.226861 3316 log.cc:1196] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:23.227155 3316 ts_tablet_manager.cc:1939] T 750ad6cb5f914983bc962da89b81cb31 P c65ed372d68e4503b01b1d7d70c8435f: Deleting consensus metadata
I20260927 18:45:23.228112 2808 catalog_manager.cc:4841] TS c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025): tablet 750ad6cb5f914983bc962da89b81cb31 (table test-table [id=7957cee3949c4f6988f2d37dff106a9b]) successfully deleted
I20260927 18:45:23.228928 3317 ts_tablet_manager.cc:1918] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:23.229012 3317 log.cc:1196] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:23.229310 3317 ts_tablet_manager.cc:1939] T 750ad6cb5f914983bc962da89b81cb31 P 5cb20d4056064b6c8b26b5500e911c03: Deleting consensus metadata
W20260927 18:45:23.230201 2809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:43461 (error 108)
I20260927 18:45:23.230288 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3015
I20260927 18:45:23.230587 2809 catalog_manager.cc:4841] TS 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): tablet 750ad6cb5f914983bc962da89b81cb31 (table test-table [id=7957cee3949c4f6988f2d37dff106a9b]) successfully deleted
I20260927 18:45:23.231673 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.231413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:43461 (local address 127.0.0.1:59094)
0927 18:45:23.231533 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.231600 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:45:23.231843 2809 catalog_manager.cc:4584] TS bbf3cda847534696ae8f14cbb07fd4d8 (127.18.235.129:43461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 750ad6cb5f914983bc962da89b81cb31: Network error: Client connection negotiation failed: client connection to 127.18.235.129:43461: connect: Connection refused (error 111)
I20260927 18:45:23.235105 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3152
I20260927 18:45:23.239430 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 2790
I20260927 18:45:43.244684 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44351
--webserver_interface=127.18.235.190
--webserver_port=46549
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:43.347640 3319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:43.347854 3319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:43.347893 3319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:43.351766 3319 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:45:43.351809 3319 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:45:43.351822 3319 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:45:43.351851 3319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:43.351882 3319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:43.351910 3319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:43.351939 3319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:43.351967 3319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:43.356873 3319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:43.357738 3319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:43.358618 3319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:43.366185 3327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:43.366302 3319 server_base.cc:1034] running on GCE node
W20260927 18:45:43.366179 3324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.366182 3325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:43.366927 3319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:43.367847 3319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:43.369001 3319 hybrid_clock.cc:648] HybridClock initialized: now 1790534743368985 us; error 34 us; skew 500 ppm
Sep 27 18:45:43 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:43.371495 3319 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:45:43.372265 3319 webserver.cc:460] Webserver started at http://127.18.235.190:46549/ using document root <none> and password file <none>
I20260927 18:45:43.372798 3319 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:43.372864 3319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:43.376276 3319 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:43.378185 3334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.379266 3319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:45:43.379367 3319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3f1c306dee292ea1dbcca79f33538e97"
server_key_iv: "dbb79b2cb835a7914e3f073c1c284e8f"
server_key_version: "encryptionkey@0"
I20260927 18:45:43.379830 3319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:43.391252 3319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:43.393961 3319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:43.394124 3319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:43.400192 3319 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:44351
I20260927 18:45:43.400425 3386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:44351 every 8 connection(s)
I20260927 18:45:43.401160 3319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:43.404953 3387 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:45:43.406443 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap starting.
I20260927 18:45:43.409729 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3319
I20260927 18:45:43.410087 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:43461
--local_ip_for_outbound_sockets=127.18.235.129
--tserver_master_addrs=127.18.235.190:44351
--webserver_port=45241
--webserver_interface=127.18.235.129
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:43.422742 3387 log.cc:824] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:43.433362 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:43.433672 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Bootstrap complete.
I20260927 18:45:43.437292 3387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:43.437670 3387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f93e5c8a66548b98d8ad3fe24e9ca2d, State: Initialized, Role: FOLLOWER
I20260927 18:45:43.438124 3387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:43.438241 3387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:43.438305 3387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:43.438397 3387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:45:43.440078 3387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:43.440363 3387 leader_election.cc:302] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f93e5c8a66548b98d8ad3fe24e9ca2d; no voters:
I20260927 18:45:43.440634 3387 leader_election.cc:288] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:45:43.440722 3391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:45:43.440901 3391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [term 3 LEADER]: Becoming Leader. State: Replica: 8f93e5c8a66548b98d8ad3fe24e9ca2d, State: Running, Role: LEADER
I20260927 18:45:43.441246 3391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } }
I20260927 18:45:43.441792 3387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:43.442463 3392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } } }
I20260927 18:45:43.442581 3392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:43.442864 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f93e5c8a66548b98d8ad3fe24e9ca2d. Latest consensus state: current_term: 3 leader_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f93e5c8a66548b98d8ad3fe24e9ca2d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 44351 } } }
I20260927 18:45:43.442999 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:43.443284 3400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:43.448051 3400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:43.448872 3400 catalog_manager.cc:1259] Loaded cluster ID: f65414235b84493e83280354f67412df
I20260927 18:45:43.448927 3400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:43.451303 3400 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:43.452056 3400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Loaded TSK: 0
I20260927 18:45:43.452142 3400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f93e5c8a66548b98d8ad3fe24e9ca2d: Loaded TSK: 1
F20260927 18:45:43.452874 3400 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f59eb84a9ad google::LogMessage::Fail() at ??:0
@ 0x7f59eb84d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f59eb84a4de google::LogMessage::Flush() at ??:0
@ 0x7f59eb84ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f59ee50e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f59ee506f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f59f61af7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f59f618999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f59f618a178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f59f61cbdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f59f61ea82b std::function<>::operator()() at ??:0
@ 0x7f59f6189b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f59f618af3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f59f6187579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f59f61cb640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f59eda9c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f59eda9cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f59eda9e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f59eda8d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f59ef0d16db start_thread at ??:0
@ 0x7f59ea34861f clone at ??:0
W20260927 18:45:43.542373 3389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:43.542659 3389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:43.542716 3389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:43.548568 3389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:43.548635 3389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:43.548717 3389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:45:43.556057 3389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:43461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=45241
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:43.557238 3389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:43.558490 3389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:43.568353 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.568360 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.568593 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:43.568594 3389 server_base.cc:1034] running on GCE node
I20260927 18:45:43.569053 3389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:43.569734 3389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:43.570936 3389 hybrid_clock.cc:648] HybridClock initialized: now 1790534743570905 us; error 31 us; skew 500 ppm
Sep 27 18:45:43 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:43.574059 3389 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:45:43.575052 3389 webserver.cc:460] Webserver started at http://127.18.235.129:45241/ using document root <none> and password file <none>
I20260927 18:45:43.575681 3389 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:43.575768 3389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:43.580435 3389 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:43.582924 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.584281 3389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:43.584400 3389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "bbf3cda847534696ae8f14cbb07fd4d8"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "3487f89bffc22fd7d1bbd4935cbf02df"
server_key_iv: "057c05f91c6c32137a701c17bbbaa6b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:43.585059 3389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:43.601400 3389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:43.604560 3389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:43.604736 3389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:43.605403 3389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:43.607246 3451 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:45:43.610952 3389 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:45:43.611019 3389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260927 18:45:43.611074 3389 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:45:43.613098 3389 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:45:43.613175 3389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:43.613423 3451 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap starting.
I20260927 18:45:43.623322 3389 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:43461
I20260927 18:45:43.623571 3558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:43461 every 8 connection(s)
I20260927 18:45:43.624385 3389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:43.625782 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3389
I20260927 18:45:43.626250 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:43025
--local_ip_for_outbound_sockets=127.18.235.130
--tserver_master_addrs=127.18.235.190:44351
--webserver_port=41095
--webserver_interface=127.18.235.130
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:43.631515 3451 log.cc:824] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:43.639845 3451 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:43.640220 3451 tablet_bootstrap.cc:492] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Bootstrap complete.
I20260927 18:45:43.641489 3451 ts_tablet_manager.cc:1397] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.010s
I20260927 18:45:43.644380 3451 raft_consensus.cc:348] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:43.644881 3451 raft_consensus.cc:729] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf3cda847534696ae8f14cbb07fd4d8, State: Initialized, Role: FOLLOWER
I20260927 18:45:43.645367 3451 consensus_queue.cc:260] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:43.646270 3451 ts_tablet_manager.cc:1428] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260927 18:45:43.680619 3561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:43.626592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:36299)
0927 18:45:43.627992 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.628047 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:43.629021 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:43.679922 (+ 50901us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":162,"threads_started":1}
W20260927 18:45:43.681185 3559 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:44351 (0 consecutive failures): Network error: Failed to ping master at 127.18.235.190:44351: Client connection negotiation failed: client connection to 127.18.235.190:44351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:45:43.681890 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.681652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:43403)
0927 18:45:43.681772 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.681835 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:45:43.682523 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.682316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:38739)
0927 18:45:43.682407 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.682473 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:45:43.682677 3559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:45:43.738499 3562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:43.738709 3562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:43.738745 3562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:43.742506 3562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:43.742545 3562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:43.742596 3562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:45:43.747422 3562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:43025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=41095
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:43.748209 3562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:43.749017 3562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:45:43.757457 3562 server_base.cc:1034] running on GCE node
W20260927 18:45:43.757416 3569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.757439 3570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.757416 3572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:43.757941 3562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:43.758487 3562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:43.759645 3562 hybrid_clock.cc:648] HybridClock initialized: now 1790534743759631 us; error 27 us; skew 500 ppm
Sep 27 18:45:43 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:43.761996 3562 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:45:43.762720 3562 webserver.cc:460] Webserver started at http://127.18.235.130:41095/ using document root <none> and password file <none>
I20260927 18:45:43.763182 3562 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:43.763242 3562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:43.766639 3562 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:43.768383 3579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.769481 3562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:43.769583 3562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "c65ed372d68e4503b01b1d7d70c8435f"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "10c75661e894666f557a6f53001319f7"
server_key_iv: "8c24f825ac8324e2d3e8ab13c3952629"
server_key_version: "encryptionkey@0"
I20260927 18:45:43.770026 3562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:43.793623 3562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:43.795970 3562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:43.796118 3562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:43.796608 3562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:43.797326 3562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:43.797369 3562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.797422 3562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:43.797454 3562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.805446 3562 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43025
I20260927 18:45:43.805531 3692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43025 every 8 connection(s)
I20260927 18:45:43.806377 3562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:43.809777 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.808178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:47839)
0927 18:45:43.808575 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.808668 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
W20260927 18:45:43.810161 3693 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:44351 (0 consecutive failures): Network error: Failed to ping master at 127.18.235.190:44351: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
I20260927 18:45:43.810662 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.810480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:38987)
0927 18:45:43.810571 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.810614 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:45:43.811195 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.811022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:59159)
0927 18:45:43.811110 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.811150 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 18:45:43.811327 3693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:45:43.815558 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3562
I20260927 18:45:43.815920 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:46847
--local_ip_for_outbound_sockets=127.18.235.131
--tserver_master_addrs=127.18.235.190:44351
--webserver_port=39899
--webserver_interface=127.18.235.131
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:43.919078 3696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:43.919277 3696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:43.919314 3696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:43.923162 3696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:43.923208 3696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:43.923259 3696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:45:43.927989 3696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:46847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=39899
--tserver_master_addrs=127.18.235.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:45:43.928748 3696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:43.929574 3696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:43.937775 3704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.937774 3702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.937772 3701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:43.937987 3696 server_base.cc:1034] running on GCE node
I20260927 18:45:43.938335 3696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:43.938854 3696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:43.940006 3696 hybrid_clock.cc:648] HybridClock initialized: now 1790534743939987 us; error 27 us; skew 500 ppm
Sep 27 18:45:43 dist-test-slave-6lr6 krb5kdc[2242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:43.942358 3696 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:45:43.943076 3696 webserver.cc:460] Webserver started at http://127.18.235.131:39899/ using document root <none> and password file <none>
I20260927 18:45:43.943533 3696 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:43.943589 3696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:43.946938 3696 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 18:45:43.948669 3711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.949713 3696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:43.949805 3696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "5cb20d4056064b6c8b26b5500e911c03"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-6lr6"
server_key: "0eeb68b89f78ed7f62f02011e51135dc"
server_key_iv: "79c5e1384ee9a189e9cf2e264768fa2f"
server_key_version: "encryptionkey@0"
I20260927 18:45:43.950398 3696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:43.966980 3696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:43.969632 3696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:43.969781 3696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:43.970258 3696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:43.970944 3696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:43.970988 3696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.971035 3696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:43.971064 3696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.978956 3696 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:46847
I20260927 18:45:43.979012 3824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:46847 every 8 connection(s)
I20260927 18:45:43.979806 3696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:43.981199 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3696
I20260927 18:45:43.983258 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.981638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:48681)
0927 18:45:43.982129 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.982244 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
W20260927 18:45:43.983629 3825 heartbeater.cc:644] Failed to heartbeat to 127.18.235.190:44351 (0 consecutive failures): Network error: Failed to ping master at 127.18.235.190:44351: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
I20260927 18:45:43.984120 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.983924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:33617)
0927 18:45:43.984021 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.984070 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:45:43.984649 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.984465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:52783)
0927 18:45:43.984560 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.984603 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:45:43.984783 3825 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:45:44.683503 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.683122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:42281)
0927 18:45:44.683360 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.683417 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":66,"threads_started":1}
I20260927 18:45:44.812076 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.811755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:50367)
0927 18:45:44.811945 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.811991 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260927 18:45:44.933350 3831 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:44.933549 3831 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:44.934568 3831 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
Sep 27 18:45:44 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.131@KRBTEST.COM
Sep 27 18:45:44 dist-test-slave-6lr6 krb5kdc[2242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.130@KRBTEST.COM
I20260927 18:45:44.950512 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.934742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43025 (local address 127.18.235.129:36653)
0927 18:45:44.934861 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.934876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.934982 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.937053 (+ 2071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.937059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.937629 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.938423 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.938444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.940254 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.940258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.941780 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.941792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.942700 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.942704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.942964 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.943608 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.943628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.945698 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:44.947621 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:44.947624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:44.947638 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:44.948103 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:44.948468 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:44.948472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:44.948474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:44.948680 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:44.949426 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:44.949432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:44.949784 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 18:45:44.949996 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 18:45:44.950308 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":309,"spinlock_wait_cycles":30592}
I20260927 18:45:44.950603 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.934976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:36653 (local address 127.18.235.130:43025)
0927 18:45:44.935374 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.935393 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.935421 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:45:44.936171 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.936181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.936633 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.936955 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.938587 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.940113 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.941891 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.942593 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.942751 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.945886 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:44.945892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:44.945906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:44.945937 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:44.947527 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:44.948212 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:44.948218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:44.948224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:44.948383 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:44.948774 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:44.948777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:44.948779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:44.949322 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:44.949452 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:44.950137 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 18:45:44.950418 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":78,"threads_started":1}
I20260927 18:45:44.950603 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.934941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:57935 (local address 127.18.235.131:46847)
0927 18:45:44.935241 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.935257 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.935297 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:45:44.936171 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.936181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.936633 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.936955 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.938587 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.940115 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.941891 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.942593 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.942751 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.945621 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:44.945626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:44.945637 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:44.945664 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:44.947469 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:44.948212 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:44.948218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:44.948224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:44.948383 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:44.948774 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:44.948777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:44.948779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:44.949322 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:44.949452 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:44.950137 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 18:45:44.950418 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":125,"threads_started":1}
I20260927 18:45:44.950520 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.934742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:46847 (local address 127.18.235.129:57935)
0927 18:45:44.935174 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.935188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.935278 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.937053 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.937059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.937629 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.938423 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.938444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.940259 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.940262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.941780 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.941792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.942700 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.942704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.942967 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.943619 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.943636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.945420 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:44.947617 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:44.947623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:44.947638 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:44.948103 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:44.948468 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:44.948472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:44.948474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:44.948680 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:44.949425 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:44.949432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:44.949784 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 18:45:44.949996 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 18:45:44.950373 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":578,"spinlock_wait_cycles":36480,"thread_start_us":64,"threads_started":1}
W20260927 18:45:44.953202 3447 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
W20260927 18:45:44.953419 3446 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:44.953486 3446 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8; no voters: 5cb20d4056064b6c8b26b5500e911c03, c65ed372d68e4503b01b1d7d70c8435f
I20260927 18:45:44.953691 3831 raft_consensus.cc:2726] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:45:44.985543 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.985219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:43195)
0927 18:45:44.985419 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.985464 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260927 18:45:45.684484 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.684118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:59437)
0927 18:45:45.684347 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.684395 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260927 18:45:45.813006 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.812663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:47009)
0927 18:45:45.812881 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.812928 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260927 18:45:45.986460 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.986076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:58013)
0927 18:45:45.986308 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.986369 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260927 18:45:46.579603 3839 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:46.579715 3839 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:46.580049 3839 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
W20260927 18:45:46.580771 3446 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
W20260927 18:45:46.580873 3447 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:46.580956 3447 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8; no voters: 5cb20d4056064b6c8b26b5500e911c03, c65ed372d68e4503b01b1d7d70c8435f
I20260927 18:45:46.581120 3839 raft_consensus.cc:2726] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:45:46.685528 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:46.685075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:39171)
0927 18:45:46.685369 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:46.685432 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260927 18:45:46.813925 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:46.813584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:60481)
0927 18:45:46.813798 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:46.813847 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260927 18:45:46.987370 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:46.987046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:47573)
0927 18:45:46.987243 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:46.987291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260927 18:45:47.686502 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:47.686106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:33561)
0927 18:45:47.686352 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:47.686405 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260927 18:45:47.814848 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:47.814491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:48959)
0927 18:45:47.814716 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:47.814767 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260927 18:45:47.988281 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:47.987947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:42315)
0927 18:45:47.988148 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:47.988196 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260927 18:45:48.687634 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:48.687246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:59937)
0927 18:45:48.687492 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:48.687541 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260927 18:45:48.815786 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:48.815436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:33345)
0927 18:45:48.815658 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:48.815707 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260927 18:45:48.989248 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:48.988869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:42757)
0927 18:45:48.989091 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:48.989169 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260927 18:45:49.167394 3849 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:49.167516 3849 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:49.167922 3849 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
W20260927 18:45:49.168650 3446 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
W20260927 18:45:49.168781 3447 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:49.168864 3447 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8; no voters: 5cb20d4056064b6c8b26b5500e911c03, c65ed372d68e4503b01b1d7d70c8435f
I20260927 18:45:49.168989 3849 raft_consensus.cc:2726] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:45:49.688670 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:49.688287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:38045)
0927 18:45:49.688523 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:49.688577 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
I20260927 18:45:49.816784 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:49.816433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:52287)
0927 18:45:49.816655 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:49.816703 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260927 18:45:49.990176 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:49.989806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:49445)
0927 18:45:49.990034 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:49.990097 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260927 18:45:50.689708 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:50.689311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:44367)
0927 18:45:50.689565 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:50.689615 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260927 18:45:50.817926 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:50.817473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:59745)
0927 18:45:50.817761 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:50.817820 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":61,"threads_started":1}
I20260927 18:45:50.991128 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:50.990751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:33651)
0927 18:45:50.990999 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:50.991048 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":63,"threads_started":1}
I20260927 18:45:51.690712 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:51.690337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:48301)
0927 18:45:51.690557 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:51.690619 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260927 18:45:51.818945 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:51.818540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:55989)
0927 18:45:51.818795 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:51.818856 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260927 18:45:51.992134 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:51.991720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:33577)
0927 18:45:51.991977 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:51.992035 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":64,"threads_started":1}
I20260927 18:45:52.691701 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:52.691336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:43929)
0927 18:45:52.691559 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:52.691609 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260927 18:45:52.819927 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:52.819585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:39947)
0927 18:45:52.819797 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:52.819846 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260927 18:45:52.993170 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:52.992772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:45245)
0927 18:45:52.993006 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:52.993056 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260927 18:45:53.182883 3862 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:53.183014 3862 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:53.183358 3862 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
W20260927 18:45:53.184056 3447 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
W20260927 18:45:53.184175 3446 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:53.184238 3446 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8; no voters: 5cb20d4056064b6c8b26b5500e911c03, c65ed372d68e4503b01b1d7d70c8435f
I20260927 18:45:53.184355 3862 raft_consensus.cc:2726] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:45:53.692761 3863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:53.692396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:47841)
0927 18:45:53.692620 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:53.692673 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260927 18:45:53.820897 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:53.820552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:47887)
0927 18:45:53.820764 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:53.820814 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260927 18:45:53.994243 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:53.993887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:51159)
0927 18:45:53.994111 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:53.994160 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260927 18:45:54.693805 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:54.693420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:56293)
0927 18:45:54.693665 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:54.693715 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
I20260927 18:45:54.821921 3867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:54.821522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:47065)
0927 18:45:54.821780 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:54.821837 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260927 18:45:54.995208 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:54.994851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:33349)
0927 18:45:54.995066 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:54.995123 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260927 18:45:55.399107 3869 raft_consensus.cc:482] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:55.399231 3869 raft_consensus.cc:504] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb20d4056064b6c8b26b5500e911c03" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 46847 } } peers { permanent_uuid: "c65ed372d68e4503b01b1d7d70c8435f" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43025 } } peers { permanent_uuid: "bbf3cda847534696ae8f14cbb07fd4d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 43461 } }
I20260927 18:45:55.399641 3869 leader_election.cc:288] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847), c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025)
W20260927 18:45:55.400374 3447 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cb20d4056064b6c8b26b5500e911c03 (127.18.235.131:46847): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
W20260927 18:45:55.400487 3446 leader_election.cc:341] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c65ed372d68e4503b01b1d7d70c8435f (127.18.235.130:43025): Not found: Tablet not found: 750ad6cb5f914983bc962da89b81cb31
I20260927 18:45:55.400583 3446 leader_election.cc:302] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbf3cda847534696ae8f14cbb07fd4d8; no voters: 5cb20d4056064b6c8b26b5500e911c03, c65ed372d68e4503b01b1d7d70c8435f
I20260927 18:45:55.400715 3869 raft_consensus.cc:2726] T 750ad6cb5f914983bc962da89b81cb31 P bbf3cda847534696ae8f14cbb07fd4d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:45:55.694772 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:55.694409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:57287)
0927 18:45:55.694632 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:55.694681 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260927 18:45:55.822901 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:55.822550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:37465)
0927 18:45:55.822765 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:55.822814 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260927 18:45:55.996163 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:55.995788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:42069)
0927 18:45:55.996016 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:55.996078 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260927 18:45:56.695819 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:56.695422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:53333)
0927 18:45:56.695675 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:56.695727 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260927 18:45:56.823962 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:56.823591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:33923)
0927 18:45:56.823825 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:56.823875 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
I20260927 18:45:56.997191 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:56.996793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:46531)
0927 18:45:56.997005 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:56.997070 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260927 18:45:57.696841 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:57.696455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:55445)
0927 18:45:57.696697 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:57.696749 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260927 18:45:57.824962 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:57.824618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:36031)
0927 18:45:57.824828 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:57.824877 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260927 18:45:57.998180 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:57.997804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.131:53731)
0927 18:45:57.998025 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:57.998088 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260927 18:45:58.698084 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:58.697699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.129:35827)
0927 18:45:58.697942 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:58.697992 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260927 18:45:58.826112 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:58.825744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:44351 (local address 127.18.235.130:57521)
0927 18:45:58.825972 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:58.826022 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.190:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 18:45:58.982451 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3389
I20260927 18:45:58.987037 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3562
I20260927 18:45:58.991433 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3696
2026-09-27T18:45:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59153 ms)
[----------] 30 tests from SecurityITest (226830 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:59 dist-test-slave-6lr6 krb5kdc[3883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:59 dist-test-slave-6lr6 krb5kdc[3883](info): set up 2 sockets
Sep 27 18:45:59 dist-test-slave-6lr6 krb5kdc[3883](info): commencing operation
krb5kdc: starting...
W20260927 18:45:59.936594 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:00.043761 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:41273
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:00.145025 3896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:00.145256 3896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:00.145293 3896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:00.149076 3896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:00.149132 3896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:00.149149 3896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:00.149164 3896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:00.149187 3896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:00.154026 3896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:00.154842 3896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:00.155692 3896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:00.163313 3896 server_base.cc:1034] running on GCE node
W20260927 18:46:00.163249 3901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.163347 3902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.163264 3904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.163887 3896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:00.164827 3896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:00.165966 3896 hybrid_clock.cc:648] HybridClock initialized: now 1790534760165946 us; error 32 us; skew 500 ppm
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:00.168370 3896 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:00.169188 3896 webserver.cc:460] Webserver started at http://127.18.235.190:44177/ using document root <none> and password file <none>
I20260927 18:46:00.169661 3896 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:00.169723 3896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:00.169888 3896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:00.171689 3896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "8579641ff8214800a01a7ccfdcb5c64a"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "014257a913e0704df528924b91946b80"
server_key_iv: "5d40b2801f9784480860f440f08ce3b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.172195 3896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "8579641ff8214800a01a7ccfdcb5c64a"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "014257a913e0704df528924b91946b80"
server_key_iv: "5d40b2801f9784480860f440f08ce3b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.175429 3896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:00.177532 3911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.178582 3896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:00.178685 3896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8579641ff8214800a01a7ccfdcb5c64a"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "014257a913e0704df528924b91946b80"
server_key_iv: "5d40b2801f9784480860f440f08ce3b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.178774 3896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:00.191620 3896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:00.194341 3896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:00.194495 3896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:00.200819 3896 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:41273
I20260927 18:46:00.200835 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:41273 every 8 connection(s)
I20260927 18:46:00.201658 3896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:00.204097 3964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:00.208056 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3896
I20260927 18:46:00.208181 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:00.208434 19374 external_mini_cluster.cc:1428] Setting key 2b687d8339ca5a67df02b861bbbe41aa
I20260927 18:46:00.209326 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: Bootstrap starting.
I20260927 18:46:00.211732 3964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:00.212504 3964 log.cc:824] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:00.214326 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: No bootstrap required, opened a new log
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:00.217986 3964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8579641ff8214800a01a7ccfdcb5c64a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41273 } }
I20260927 18:46:00.218250 3964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:00.218315 3964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8579641ff8214800a01a7ccfdcb5c64a, State: Initialized, Role: FOLLOWER
I20260927 18:46:00.218766 3964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8579641ff8214800a01a7ccfdcb5c64a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41273 } }
I20260927 18:46:00.218917 3964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:00.218981 3964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:00.219066 3964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:00.219949 3964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8579641ff8214800a01a7ccfdcb5c64a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41273 } }
I20260927 18:46:00.220285 3964 leader_election.cc:302] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8579641ff8214800a01a7ccfdcb5c64a; no voters:
I20260927 18:46:00.220573 3964 leader_election.cc:288] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:00.220647 3969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:00.220846 3969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [term 1 LEADER]: Becoming Leader. State: Replica: 8579641ff8214800a01a7ccfdcb5c64a, State: Running, Role: LEADER
I20260927 18:46:00.221204 3969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8579641ff8214800a01a7ccfdcb5c64a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41273 } }
I20260927 18:46:00.221745 3964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:00.222271 3970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8579641ff8214800a01a7ccfdcb5c64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8579641ff8214800a01a7ccfdcb5c64a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41273 } } }
I20260927 18:46:00.222379 3970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:00.222383 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.209673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.18.235.190:41273)
0927 18:46:00.210032 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.210052 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.210091 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:46:00.210900 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.210921 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.211296 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.211623 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.212279 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.213564 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.214400 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.215053 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.215212 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.217559 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.217565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.217579 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:00.217608 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.219436 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.219823 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.219828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.219834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.219966 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.220229 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.220232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.220233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.220710 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.220891 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.221192 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.221502 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
I20260927 18:46:00.222985 3971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8579641ff8214800a01a7ccfdcb5c64a. Latest consensus state: current_term: 1 leader_uuid: "8579641ff8214800a01a7ccfdcb5c64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8579641ff8214800a01a7ccfdcb5c64a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41273 } } }
I20260927 18:46:00.223064 3971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:00.226516 3984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:00.226577 3984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:00.226660 3975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:00.227336 3975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:00.232120 3975 catalog_manager.cc:1347] Generated new cluster ID: 74dde4167d9b49c9a1075d1e02170b90
I20260927 18:46:00.232183 3975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:00.283064 3975 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:00.284281 3975 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:00.300874 3975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: Generated new TSK 0
I20260927 18:46:00.301671 3975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:46:00.337821 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41273
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:00.443552 3990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:00.443776 3990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:00.443816 3990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:00.447687 3990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:00.447732 3990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:00.447793 3990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:46:00.452741 3990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:00.453626 3990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:00.454514 3990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:00.463140 3998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.463131 3995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.463112 3996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.463392 3990 server_base.cc:1034] running on GCE node
I20260927 18:46:00.463804 3990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:00.464352 3990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:00.465497 3990 hybrid_clock.cc:648] HybridClock initialized: now 1790534760465469 us; error 41 us; skew 500 ppm
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:00.467957 3990 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:46:00.468726 3990 webserver.cc:460] Webserver started at http://127.18.235.129:34151/ using document root <none> and password file <none>
I20260927 18:46:00.469231 3990 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:00.469291 3990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:00.469444 3990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:00.471237 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "1d8790cfa2d8451ab7d5e393c821b8e6"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "d766c916c52466e94ed523254e80d16c"
server_key_iv: "533ad3fcff6853a3ed807061140cbb5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.471697 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "1d8790cfa2d8451ab7d5e393c821b8e6"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "d766c916c52466e94ed523254e80d16c"
server_key_iv: "533ad3fcff6853a3ed807061140cbb5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.474931 3990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:00.477001 4005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.478094 3990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:00.478181 3990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "1d8790cfa2d8451ab7d5e393c821b8e6"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "d766c916c52466e94ed523254e80d16c"
server_key_iv: "533ad3fcff6853a3ed807061140cbb5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.478273 3990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:00.503981 3990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:00.506520 3990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:00.506659 3990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:00.507149 3990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:00.507838 3990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:00.507881 3990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.507921 3990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:00.507951 3990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.515936 3990 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:33461
I20260927 18:46:00.515964 4118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:33461 every 8 connection(s)
I20260927 18:46:00.516701 3990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:00.518915 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 3990
I20260927 18:46:00.519054 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:00.519392 19374 external_mini_cluster.cc:1428] Setting key fd4ce33cef0e4cc364ff090f64aafb46
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:46:00.530166 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.518412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:42337 (local address 127.18.235.190:41273)
0927 18:46:00.518555 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.518559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.519374 (+ 815us) server_negotiation.cc:407] Connection header received
0927 18:46:00.520693 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.520696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.520745 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.520826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.521896 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.522658 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.524067 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.524424 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.524541 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.526984 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.526987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.526990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:00.527013 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.528274 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.528852 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.528854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.528855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.528925 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.529284 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.529286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.529287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.529437 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.529507 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.529957 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.530050 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:46:00.530918 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.518725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41273 (local address 127.18.235.129:42337)
0927 18:46:00.519241 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.519278 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.520503 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.520960 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.520966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.521294 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:00.521757 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.521771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.522775 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.522779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.523943 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.523953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.524552 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.524554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.524754 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.525330 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:00.525344 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:00.526821 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:00.528391 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:00.528397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:00.528411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:00.528756 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:00.529011 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:00.529013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:00.529014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:00.529206 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:00.529515 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:00.529521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:00.529760 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.529907 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.530149 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
I20260927 18:46:00.532126 4119 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41273
I20260927 18:46:00.532331 4119 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:00.532768 4119 heartbeater.cc:511] Master 127.18.235.190:41273 requested a full tablet report, sending...
I20260927 18:46:00.534166 3928 ts_manager.cc:194] Registered new tserver with Master: 1d8790cfa2d8451ab7d5e393c821b8e6 (127.18.235.129:33461)
I20260927 18:46:00.535310 3928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:42337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:46:00.547555 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41273
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:00.647650 4124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:00.647861 4124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:00.647913 4124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:00.651697 4124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:00.651744 4124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:00.651819 4124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:46:00.656708 4124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:00.657582 4124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:00.658442 4124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:00.667055 4129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.667084 4130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.667084 4132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.667320 4124 server_base.cc:1034] running on GCE node
I20260927 18:46:00.667688 4124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:00.668270 4124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:00.669425 4124 hybrid_clock.cc:648] HybridClock initialized: now 1790534760669411 us; error 25 us; skew 500 ppm
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:00.671903 4124 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:46:00.672673 4124 webserver.cc:460] Webserver started at http://127.18.235.130:40531/ using document root <none> and password file <none>
I20260927 18:46:00.673193 4124 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:00.673266 4124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:00.673456 4124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:00.675310 4124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "4f91b766e0f245efb995552f7aeb3577"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "0cde04afef4e14f56afacc265f82c0c2"
server_key_iv: "13b5f06e82b18604efdc4a1872ffdc5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.675848 4124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "4f91b766e0f245efb995552f7aeb3577"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "0cde04afef4e14f56afacc265f82c0c2"
server_key_iv: "13b5f06e82b18604efdc4a1872ffdc5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.679327 4124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:00.681666 4139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.682837 4124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:00.682940 4124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "4f91b766e0f245efb995552f7aeb3577"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "0cde04afef4e14f56afacc265f82c0c2"
server_key_iv: "13b5f06e82b18604efdc4a1872ffdc5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.683034 4124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:00.696105 4124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:00.698770 4124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:00.698943 4124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:00.699455 4124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:00.700217 4124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:00.700268 4124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.700320 4124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:00.700354 4124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.708851 4124 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43305
I20260927 18:46:00.708870 4252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43305 every 8 connection(s)
I20260927 18:46:00.709671 4124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:00.712172 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4124
I20260927 18:46:00.712309 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:00.712675 19374 external_mini_cluster.cc:1428] Setting key 26f42e85c5643edf40d0e60c75a8eae8
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:46:00.723544 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.711221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:40535 (local address 127.18.235.190:41273)
0927 18:46:00.711353 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.711357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.712134 (+ 777us) server_negotiation.cc:407] Connection header received
0927 18:46:00.713712 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.713715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.713760 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.713840 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.715218 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.715780 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.716901 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.717187 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.717281 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.719758 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.719762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.719765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:00.719798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.721356 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.721976 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.721980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.721982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.722084 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.722397 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.722401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.722403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.722644 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.722762 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.723284 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.723406 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:46:00.724327 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.711519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41273 (local address 127.18.235.130:40535)
0927 18:46:00.711949 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.712003 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.713514 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.714004 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.714013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.714453 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:00.715065 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.715084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.715901 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.715904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.716784 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.716796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.717326 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.717330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.717508 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.718017 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:00.718046 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:00.719581 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:00.721479 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:00.721485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:00.721500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:00.721806 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:00.722176 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:00.722178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:00.722180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:00.722303 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:00.722766 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:00.722771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:00.723030 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.723220 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.723443 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260927 18:46:00.725425 4253 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
I20260927 18:46:00.725683 4253 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:00.726099 4253 heartbeater.cc:511] Master 127.18.235.190:41273 requested a full tablet report, sending...
I20260927 18:46:00.727066 3928 ts_manager.cc:194] Registered new tserver with Master: 4f91b766e0f245efb995552f7aeb3577 (127.18.235.130:43305)
I20260927 18:46:00.727783 3928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:40535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:46:00.739055 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41273
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:00.846860 4258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:00.847155 4258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:00.847232 4258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:00.853188 4258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:00.853241 4258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:00.853305 4258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:46:00.860613 4258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:00.861858 4258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:00.863140 4258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:00.874383 4264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.874364 4263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.874354 4266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.874980 4258 server_base.cc:1034] running on GCE node
I20260927 18:46:00.875355 4258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:00.875913 4258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:00.877086 4258 hybrid_clock.cc:648] HybridClock initialized: now 1790534760877061 us; error 37 us; skew 500 ppm
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:00.879561 4258 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:46:00.880373 4258 webserver.cc:460] Webserver started at http://127.18.235.131:43847/ using document root <none> and password file <none>
I20260927 18:46:00.880865 4258 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:00.880931 4258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:00.881156 4258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:00.883128 4258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "d0a7711ea79e4f87988ea65ded7c7a02"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "ecbdb00a01caa32c58c570c87c9024fe"
server_key_iv: "390ca846b0d8be38612eab9801355538"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.883654 4258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "d0a7711ea79e4f87988ea65ded7c7a02"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "ecbdb00a01caa32c58c570c87c9024fe"
server_key_iv: "390ca846b0d8be38612eab9801355538"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.887102 4258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 18:46:00.889258 4273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.890336 4258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:00.890439 4258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "d0a7711ea79e4f87988ea65ded7c7a02"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-6lr6"
server_key: "ecbdb00a01caa32c58c570c87c9024fe"
server_key_iv: "390ca846b0d8be38612eab9801355538"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.890530 4258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:00.917253 4258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:00.919620 4258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:00.919785 4258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:00.920293 4258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:00.921027 4258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:00.921077 4258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.921152 4258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:00.921186 4258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.929998 4258 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:41135
I20260927 18:46:00.930022 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:41135 every 8 connection(s)
I20260927 18:46:00.930778 4258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:00.933882 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4258
I20260927 18:46:00.933984 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:00.934229 19374 external_mini_cluster.cc:1428] Setting key c6979a202be0890672ef5ae256ba0ed4
Sep 27 18:46:00 dist-test-slave-6lr6 krb5kdc[3883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:00.943230 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.932324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:48509 (local address 127.18.235.190:41273)
0927 18:46:00.932534 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.932538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.933195 (+ 657us) server_negotiation.cc:407] Connection header received
0927 18:46:00.934611 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.934615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.934672 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.934754 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.935892 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.936437 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.937607 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.937824 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.937920 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.940117 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.940121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.940123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:00.940145 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.941309 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.941801 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.941806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.941808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.941887 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.942207 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.942211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.942213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.942387 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.942480 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.942934 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.943063 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 18:46:00.943862 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.932627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41273 (local address 127.18.235.131:48509)
0927 18:46:00.933043 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.933082 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.934396 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.934912 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.934919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.935280 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:00.935753 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.935766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.936570 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.936573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.937477 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.937487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.937958 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.937963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.938118 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.938609 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:00.938625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:00.939957 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:00.941422 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:00.941428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:00.941441 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:00.941701 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:00.941974 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:00.941977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:00.941979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:00.942107 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:00.942492 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:00.942499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:00.942713 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.942876 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.943165 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260927 18:46:00.944809 4387 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41273
I20260927 18:46:00.944996 4387 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:00.945418 4387 heartbeater.cc:511] Master 127.18.235.190:41273 requested a full tablet report, sending...
I20260927 18:46:00.946303 3928 ts_manager.cc:194] Registered new tserver with Master: d0a7711ea79e4f87988ea65ded7c7a02 (127.18.235.131:41135)
I20260927 18:46:00.946920 3928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:48509
I20260927 18:46:00.947571 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:00.956142 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.949416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.18.235.190:41273)
0927 18:46:00.949530 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.949535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.949721 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:46:00.949833 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.949837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.949880 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.949947 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.950633 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.951114 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.951959 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.952175 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.952267 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.953500 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.953505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.953507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:00.953531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.954561 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.954957 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.954961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.954963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.955054 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:00.955314 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:00.955318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:00.955320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:00.955517 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.955642 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.955880 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.955981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:46:00.958894 3927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:00.961730 3927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:00.974598 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.969142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.18.235.130:43305)
0927 18:46:00.969579 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.969591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.969614 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:46:00.970193 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.970199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.970403 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.970522 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:00.971082 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.971782 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.973216 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.973926 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.973998 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.974081 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.974396 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.974482 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":147,"threads_started":1}
I20260927 18:46:00.974594 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.968958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43305 (local address 127.0.0.1:37300)
0927 18:46:00.969397 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.969443 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.970016 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.970701 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.970704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.970737 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:00.970931 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.970937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.971903 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.971907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.973048 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.973058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.974153 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.974155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.974237 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.974260 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.974341 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.974450 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":58,"spinlock_wait_cycles":7424,"thread_start_us":51,"threads_started":1}
I20260927 18:46:00.974910 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.969067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:41135 (local address 127.0.0.1:38812)
0927 18:46:00.969586 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.969598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.969677 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.970829 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.970832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.970855 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:00.970985 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.970992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.972351 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.972354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.973542 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.973551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.974533 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.974536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.974634 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.974655 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.974704 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.974781 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":66,"thread_start_us":106,"threads_started":1}
I20260927 18:46:00.975061 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.969617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33461 (local address 127.0.0.1:40064)
0927 18:46:00.970025 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.970037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.970141 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.970698 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.970703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.970739 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:00.970916 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.970923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.972375 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.972378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.973444 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.973453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.974709 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.974712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.974816 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.974836 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.974894 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.974953 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":34,"spinlock_wait_cycles":6784,"thread_start_us":109,"threads_started":1}
I20260927 18:46:00.975419 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.969770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.18.235.129:33461)
0927 18:46:00.970166 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.970177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.970197 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:46:00.970256 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.970261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.970398 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.970527 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:00.971020 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.972235 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.973561 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.974400 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.974519 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.974641 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.975195 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.975289 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":116,"threads_started":1}
I20260927 18:46:00.975600 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.969797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.18.235.131:41135)
0927 18:46:00.970450 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.970461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.970481 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:46:00.970548 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.970553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.970705 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.970829 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:00.971104 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.972215 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.973642 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.974404 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.975248 (+ 844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.975359 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.975402 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.975482 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":95,"threads_started":1}
I20260927 18:46:00.977695 4187 tablet_service.cc:1467] Processing CreateTablet for tablet 3d50a4263b274e11a783d6276324c5d7 (DEFAULT_TABLE table=test-table [id=2485bf85dc98407f837b259506be3eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:00.978550 4187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d50a4263b274e11a783d6276324c5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:00.978271 4321 tablet_service.cc:1467] Processing CreateTablet for tablet 3d50a4263b274e11a783d6276324c5d7 (DEFAULT_TABLE table=test-table [id=2485bf85dc98407f837b259506be3eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:00.978271 4053 tablet_service.cc:1467] Processing CreateTablet for tablet 3d50a4263b274e11a783d6276324c5d7 (DEFAULT_TABLE table=test-table [id=2485bf85dc98407f837b259506be3eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:00.979243 4321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d50a4263b274e11a783d6276324c5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:00.979247 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d50a4263b274e11a783d6276324c5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:00.984944 4404 tablet_bootstrap.cc:492] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Bootstrap starting.
I20260927 18:46:00.985895 4405 tablet_bootstrap.cc:492] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: Bootstrap starting.
I20260927 18:46:00.987159 4404 tablet_bootstrap.cc:654] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:00.987989 4405 tablet_bootstrap.cc:654] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:00.988404 4404 log.cc:824] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:00.989130 4405 log.cc:824] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:00.989387 4406 tablet_bootstrap.cc:492] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Bootstrap starting.
I20260927 18:46:00.990342 4404 tablet_bootstrap.cc:492] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: No bootstrap required, opened a new log
I20260927 18:46:00.990525 4404 ts_tablet_manager.cc:1397] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:46:00.990947 4405 tablet_bootstrap.cc:492] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: No bootstrap required, opened a new log
I20260927 18:46:00.991124 4405 ts_tablet_manager.cc:1397] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:00.991214 4406 tablet_bootstrap.cc:654] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:00.992134 4406 log.cc:824] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:00.993664 4406 tablet_bootstrap.cc:492] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: No bootstrap required, opened a new log
I20260927 18:46:00.993863 4406 ts_tablet_manager.cc:1397] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:00.994264 4404 raft_consensus.cc:348] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:00.994632 4404 raft_consensus.cc:374] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:00.994699 4404 raft_consensus.cc:729] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f91b766e0f245efb995552f7aeb3577, State: Initialized, Role: FOLLOWER
I20260927 18:46:00.994630 4405 raft_consensus.cc:348] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:00.994884 4405 raft_consensus.cc:374] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:00.994954 4405 raft_consensus.cc:729] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8790cfa2d8451ab7d5e393c821b8e6, State: Initialized, Role: FOLLOWER
I20260927 18:46:00.995189 4404 consensus_queue.cc:260] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:00.995319 4405 consensus_queue.cc:260] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:00.995898 4119 heartbeater.cc:503] Master 127.18.235.190:41273 was elected leader, sending a full tablet report...
I20260927 18:46:00.996117 4404 ts_tablet_manager.cc:1428] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:46:00.996428 4253 heartbeater.cc:503] Master 127.18.235.190:41273 was elected leader, sending a full tablet report...
I20260927 18:46:00.997368 4405 ts_tablet_manager.cc:1428] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:46:00.997233 4406 raft_consensus.cc:348] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:00.998023 4406 raft_consensus.cc:374] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:00.998082 4406 raft_consensus.cc:729] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a7711ea79e4f87988ea65ded7c7a02, State: Initialized, Role: FOLLOWER
I20260927 18:46:00.998490 4406 consensus_queue.cc:260] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:00.999377 4406 ts_tablet_manager.cc:1428] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:00.999817 4387 heartbeater.cc:503] Master 127.18.235.190:41273 was elected leader, sending a full tablet report...
W20260927 18:46:01.017681 4120 tablet.cc:2369] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:01.174887 4411 raft_consensus.cc:482] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:01.175093 4411 raft_consensus.cc:504] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:01.176185 4411 leader_election.cc:288] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f91b766e0f245efb995552f7aeb3577 (127.18.235.130:43305), d0a7711ea79e4f87988ea65ded7c7a02 (127.18.235.131:41135)
I20260927 18:46:01.179842 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.176504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:57767 (local address 127.18.235.130:43305)
0927 18:46:01.176659 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.176665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.176799 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:46:01.176896 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.176899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.176950 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.177017 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:01.177341 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.177844 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.178939 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.179316 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.179419 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.179468 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.179649 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.179717 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:46:01.179865 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.176545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:54021 (local address 127.18.235.131:41135)
0927 18:46:01.176675 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.176680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.176890 (+ 210us) server_negotiation.cc:407] Connection header received
0927 18:46:01.176986 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.176989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.177042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.177142 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:01.177372 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.177859 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.178937 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.179314 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.179413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.179463 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.179695 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.179760 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:46:01.179875 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.176444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:41135 (local address 127.18.235.129:54021)
0927 18:46:01.176790 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.176803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:01.176902 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:01.177144 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:01.177147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:01.177163 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:01.177267 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.177272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.177959 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.177963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.178815 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.178824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.179420 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.179423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.179521 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.179587 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 18:46:01.179657 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:46:01.179720 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":73,"thread_start_us":109,"threads_started":1}
I20260927 18:46:01.179883 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.176415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43305 (local address 127.18.235.129:57767)
0927 18:46:01.176716 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.176728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:01.176818 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:01.177042 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:01.177045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:01.177060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:01.177226 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.177234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.177946 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.177949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.178820 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.178828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.179421 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.179425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.179544 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.179562 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:46:01.179615 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:46:01.179678 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260927 18:46:01.180433 4341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d50a4263b274e11a783d6276324c5d7" candidate_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" is_pre_election: true
I20260927 18:46:01.180594 4207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d50a4263b274e11a783d6276324c5d7" candidate_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f91b766e0f245efb995552f7aeb3577" is_pre_election: true
I20260927 18:46:01.180758 4341 raft_consensus.cc:2454] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8790cfa2d8451ab7d5e393c821b8e6 in term 0.
I20260927 18:46:01.180837 4207 raft_consensus.cc:2454] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8790cfa2d8451ab7d5e393c821b8e6 in term 0.
I20260927 18:46:01.181277 4008 leader_election.cc:302] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8790cfa2d8451ab7d5e393c821b8e6, 4f91b766e0f245efb995552f7aeb3577; no voters:
W20260927 18:46:01.181615 4388 tablet.cc:2369] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:01.181710 4411 raft_consensus.cc:2781] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:01.181810 4411 raft_consensus.cc:482] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:01.181876 4411 raft_consensus.cc:3037] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:01.182956 4411 raft_consensus.cc:504] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:01.183254 4411 leader_election.cc:288] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [CANDIDATE]: Term 1 election: Requested vote from peers 4f91b766e0f245efb995552f7aeb3577 (127.18.235.130:43305), d0a7711ea79e4f87988ea65ded7c7a02 (127.18.235.131:41135)
I20260927 18:46:01.183535 4207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d50a4263b274e11a783d6276324c5d7" candidate_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f91b766e0f245efb995552f7aeb3577"
I20260927 18:46:01.183584 4341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d50a4263b274e11a783d6276324c5d7" candidate_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a7711ea79e4f87988ea65ded7c7a02"
I20260927 18:46:01.183640 4207 raft_consensus.cc:3037] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:01.183704 4341 raft_consensus.cc:3037] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:01.184612 4207 raft_consensus.cc:2454] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8790cfa2d8451ab7d5e393c821b8e6 in term 1.
I20260927 18:46:01.184706 4341 raft_consensus.cc:2454] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8790cfa2d8451ab7d5e393c821b8e6 in term 1.
I20260927 18:46:01.184909 4008 leader_election.cc:302] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8790cfa2d8451ab7d5e393c821b8e6, 4f91b766e0f245efb995552f7aeb3577; no voters:
I20260927 18:46:01.185045 4411 raft_consensus.cc:2781] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:01.185256 4411 raft_consensus.cc:686] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8790cfa2d8451ab7d5e393c821b8e6, State: Running, Role: LEADER
I20260927 18:46:01.185565 4411 consensus_queue.cc:237] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } }
I20260927 18:46:01.188506 3927 catalog_manager.cc:5495] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d8790cfa2d8451ab7d5e393c821b8e6 (127.18.235.129). New cstate: current_term: 1 leader_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d8790cfa2d8451ab7d5e393c821b8e6" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33461 } health_report { overall_health: HEALTHY } } }
I20260927 18:46:01.204175 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.200588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.18.235.129:33461)
0927 18:46:01.200707 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.200712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.200727 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:01.200801 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.200804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.200862 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.200964 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:01.201195 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.201941 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.202773 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.203099 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.203190 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.203411 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:01.203876 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:01.203979 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.204006 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.204059 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:01.209896 4207 raft_consensus.cc:1261] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 1 FOLLOWER]: Refusing update from remote peer 1d8790cfa2d8451ab7d5e393c821b8e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:01.209896 4341 raft_consensus.cc:1261] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 1 FOLLOWER]: Refusing update from remote peer 1d8790cfa2d8451ab7d5e393c821b8e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:01.210614 4411 consensus_queue.cc:1035] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f91b766e0f245efb995552f7aeb3577" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:46:01.210855 4254 tablet.cc:2369] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:01.210851 4415 consensus_queue.cc:1035] T 3d50a4263b274e11a783d6276324c5d7 P 1d8790cfa2d8451ab7d5e393c821b8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a7711ea79e4f87988ea65ded7c7a02" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:01.218004 4419 mvcc.cc:205] Tried to move back new op lower bound from 7334030381909131264 to 7334030381818306560. Current Snapshot: MvccSnapshot[applied={T|T < 7334030381909131264}]
I20260927 18:46:01.225571 3927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:01.225697 3927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:01.227715 3927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8579641ff8214800a01a7ccfdcb5c64a: Sending DeleteTablet for 3 replicas of tablet 3d50a4263b274e11a783d6276324c5d7
I20260927 18:46:01.228345 4187 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d50a4263b274e11a783d6276324c5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:01 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:37300
I20260927 18:46:01.228497 4321 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d50a4263b274e11a783d6276324c5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:01 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:38812
I20260927 18:46:01.228739 4428 tablet_replica.cc:330] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: stopping tablet replica
I20260927 18:46:01.228754 4053 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d50a4263b274e11a783d6276324c5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:01 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:40064
I20260927 18:46:01.228996 4428 raft_consensus.cc:2229] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:01.229288 4428 raft_consensus.cc:2258] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:01.229319 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3990
I20260927 18:46:01.230988 4429 tablet_replica.cc:330] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: stopping tablet replica
I20260927 18:46:01.231228 4429 raft_consensus.cc:2229] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:01.231288 4428 ts_tablet_manager.cc:1905] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:01.231462 4429 raft_consensus.cc:2258] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:01.232605 4429 ts_tablet_manager.cc:1905] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:01.233819 4428 ts_tablet_manager.cc:1918] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:01.233923 4428 log.cc:1196] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/3d50a4263b274e11a783d6276324c5d7
I20260927 18:46:01.234213 4428 ts_tablet_manager.cc:1939] T 3d50a4263b274e11a783d6276324c5d7 P 4f91b766e0f245efb995552f7aeb3577: Deleting consensus metadata
I20260927 18:46:01.234951 4429 ts_tablet_manager.cc:1918] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:01.235036 4429 log.cc:1196] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/3d50a4263b274e11a783d6276324c5d7
I20260927 18:46:01.235322 4429 ts_tablet_manager.cc:1939] T 3d50a4263b274e11a783d6276324c5d7 P d0a7711ea79e4f87988ea65ded7c7a02: Deleting consensus metadata
I20260927 18:46:01.235311 3914 catalog_manager.cc:4841] TS 4f91b766e0f245efb995552f7aeb3577 (127.18.235.130:43305): tablet 3d50a4263b274e11a783d6276324c5d7 (table test-table [id=2485bf85dc98407f837b259506be3eda]) successfully deleted
W20260927 18:46:01.235558 3912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:33461 (error 108)
I20260927 18:46:01.235718 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4124
I20260927 18:46:01.236470 3912 catalog_manager.cc:4841] TS d0a7711ea79e4f87988ea65ded7c7a02 (127.18.235.131:41135): tablet 3d50a4263b274e11a783d6276324c5d7 (table test-table [id=2485bf85dc98407f837b259506be3eda]) successfully deleted
I20260927 18:46:01.236588 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.236259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33461 (local address 127.0.0.1:40088)
0927 18:46:01.236397 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.236462 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:33461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 18:46:01.236712 3912 catalog_manager.cc:4584] TS 1d8790cfa2d8451ab7d5e393c821b8e6 (127.18.235.129:33461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d50a4263b274e11a783d6276324c5d7: Network error: Client connection negotiation failed: client connection to 127.18.235.129:33461: connect: Connection refused (error 111)
I20260927 18:46:01.241714 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4258
I20260927 18:46:01.246392 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 3896
2026-09-27T18:46:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2254 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T18:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:01Z Disabled control of system clock
I20260927 18:46:01.272374 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38667
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:38667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:46:01.372673 4435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:01.372875 4435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:01.372906 4435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:01.376830 4435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:01.376875 4435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:01.376889 4435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:01.376904 4435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:01.376917 4435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:01.381417 4435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:38667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:01.382261 4435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:01.383446 4435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:01.391126 4443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.391134 4440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.391284 4435 server_base.cc:1034] running on GCE node
W20260927 18:46:01.391136 4441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.391855 4435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:01.392807 4435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:01.393953 4435 hybrid_clock.cc:648] HybridClock initialized: now 1790534761393935 us; error 33 us; skew 500 ppm
I20260927 18:46:01.395542 4435 webserver.cc:460] Webserver started at http://127.18.235.190:35603/ using document root <none> and password file <none>
I20260927 18:46:01.396042 4435 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:01.396103 4435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:01.396288 4435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:01.398151 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "8efa855551f64467ba2976c434e809d8"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "3c1bce49f2f2e9fb5f15a401ce3f3a41"
server_key_iv: "95c31bc223f22fdfcbaad50eaa23c944"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.398634 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "8efa855551f64467ba2976c434e809d8"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "3c1bce49f2f2e9fb5f15a401ce3f3a41"
server_key_iv: "95c31bc223f22fdfcbaad50eaa23c944"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.401871 4435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:01.404026 4449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.405077 4435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:01.405205 4435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "8efa855551f64467ba2976c434e809d8"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "3c1bce49f2f2e9fb5f15a401ce3f3a41"
server_key_iv: "95c31bc223f22fdfcbaad50eaa23c944"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.405290 4435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:01.422434 4435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:01.422977 4435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:01.423126 4435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:01.429184 4435 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:38667
I20260927 18:46:01.429214 4501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:38667 every 8 connection(s)
I20260927 18:46:01.429986 4435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:01.432796 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:01.436942 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4435
I20260927 18:46:01.437067 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:01.437356 19374 external_mini_cluster.cc:1428] Setting key 1631e463d8d8c3d1753f8e2be415106b
I20260927 18:46:01.438328 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8: Bootstrap starting.
I20260927 18:46:01.440480 4502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:01.441191 4502 log.cc:824] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:01.442853 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8: No bootstrap required, opened a new log
I20260927 18:46:01.445868 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.438715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.18.235.190:38667)
0927 18:46:01.439312 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.439326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.439359 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:46:01.439938 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.439954 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.440328 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.440613 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.441033 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.442026 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.442875 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.443386 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.443528 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.444274 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.444281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.444293 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.444638 (+ 345us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.444692 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.444710 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.444792 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.444931 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.445219 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260927 18:46:01.446163 4502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } }
I20260927 18:46:01.446359 4502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:01.446413 4502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efa855551f64467ba2976c434e809d8, State: Initialized, Role: FOLLOWER
I20260927 18:46:01.446849 4502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } }
I20260927 18:46:01.447037 4507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:01.447168 4507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } }
I20260927 18:46:01.447013 4502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:01.447476 4507 leader_election.cc:302] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8efa855551f64467ba2976c434e809d8; no voters:
I20260927 18:46:01.447510 4502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:01.447572 4502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:01.447779 4507 leader_election.cc:288] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 18:46:01.448367 4502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } }
I20260927 18:46:01.448474 4502 leader_election.cc:302] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8efa855551f64467ba2976c434e809d8; no voters:
I20260927 18:46:01.448523 4508 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 18:46:01.448570 4502 leader_election.cc:288] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:01.448741 4507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:01.448832 4507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8efa855551f64467ba2976c434e809d8, State: Running, Role: LEADER
I20260927 18:46:01.449085 4507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } }
I20260927 18:46:01.449565 4502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:01.450098 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8efa855551f64467ba2976c434e809d8. Latest consensus state: current_term: 1 leader_uuid: "8efa855551f64467ba2976c434e809d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } } }
I20260927 18:46:01.450228 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:01.450398 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8efa855551f64467ba2976c434e809d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa855551f64467ba2976c434e809d8" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38667 } } }
I20260927 18:46:01.450481 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:01.450606 4514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:01.453804 4514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:01.458881 4514 catalog_manager.cc:1347] Generated new cluster ID: 5d54fdc672b144dd9dc2820d035a1dc2
I20260927 18:46:01.458943 4514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:01.476418 4514 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:01.477200 4514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:01.483368 4514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8: Generated new TSK 0
I20260927 18:46:01.483922 4514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:01.495764 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38667
--builtin_ntp_servers=127.18.235.148:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:46:01.597818 4526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:01.598019 4526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:01.598052 4526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:01.601886 4526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:01.601926 4526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:01.601979 4526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:46:01.606534 4526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:01.607360 4526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:01.608582 4526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:01.617018 4531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.617117 4532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.617117 4534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.617343 4526 server_base.cc:1034] running on GCE node
I20260927 18:46:01.617693 4526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:01.618248 4526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:01.619400 4526 hybrid_clock.cc:648] HybridClock initialized: now 1790534761619386 us; error 24 us; skew 500 ppm
I20260927 18:46:01.621009 4526 webserver.cc:460] Webserver started at http://127.18.235.129:41769/ using document root <none> and password file <none>
I20260927 18:46:01.621513 4526 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:01.621577 4526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:01.621753 4526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:01.623531 4526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "13b8e83b72f64d8cb4d42f8b8d970195"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "c10870848dac4a341a027bef1c336371"
server_key_iv: "42fbbc765114b198bea82c6419815a83"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.624022 4526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "13b8e83b72f64d8cb4d42f8b8d970195"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "c10870848dac4a341a027bef1c336371"
server_key_iv: "42fbbc765114b198bea82c6419815a83"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.627219 4526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:01.629285 4540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.630335 4526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:01.630431 4526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "13b8e83b72f64d8cb4d42f8b8d970195"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "c10870848dac4a341a027bef1c336371"
server_key_iv: "42fbbc765114b198bea82c6419815a83"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.630515 4526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:01.647324 4526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:01.647851 4526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:01.647998 4526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:01.648474 4526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:01.649323 4526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:01.649370 4526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.649417 4526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:01.649446 4526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.657276 4526 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:33939
I20260927 18:46:01.657316 4653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:33939 every 8 connection(s)
I20260927 18:46:01.658033 4526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:01.661235 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4526
I20260927 18:46:01.661331 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:01.661564 19374 external_mini_cluster.cc:1428] Setting key eb225aaea786601e302851c53619495b
I20260927 18:46:01.663393 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38667
--builtin_ntp_servers=127.18.235.148:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:46:01.666483 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.659565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:53293 (local address 127.18.235.190:38667)
0927 18:46:01.659718 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.659723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.660622 (+ 899us) server_negotiation.cc:407] Connection header received
0927 18:46:01.661950 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.661963 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.662019 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.662093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.662960 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.663706 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.664778 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.664995 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.665083 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.666050 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.666054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.666056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.666066 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.666085 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.666092 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.666149 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.666290 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.666350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:46:01.667268 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.659920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38667 (local address 127.18.235.129:53293)
0927 18:46:01.660485 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.660520 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:01.661754 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:01.662264 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:01.662270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:01.662347 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:01.662833 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.662847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.663822 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.663825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.664654 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.664663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.665158 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.665163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.665303 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.665817 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:01.665819 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:01.665853 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:01.665909 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:01.666162 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:01.666170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:01.666176 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:01.666276 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:46:01.666522 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260927 18:46:01.668289 4654 heartbeater.cc:347] Connected to a master server at 127.18.235.190:38667
I20260927 18:46:01.668480 4654 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:01.668906 4654 heartbeater.cc:511] Master 127.18.235.190:38667 requested a full tablet report, sending...
I20260927 18:46:01.670220 4466 ts_manager.cc:194] Registered new tserver with Master: 13b8e83b72f64d8cb4d42f8b8d970195 (127.18.235.129:33939)
I20260927 18:46:01.671285 4466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.129:53293
W20260927 18:46:01.765869 4657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:01.766083 4657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:01.766131 4657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:01.770026 4657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:01.770069 4657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:01.770139 4657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:46:01.774569 4657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:01.775466 4657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:01.776715 4657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:01.785252 4662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.785318 4665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.785290 4657 server_base.cc:1034] running on GCE node
W20260927 18:46:01.785403 4663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.785836 4657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:01.786391 4657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:01.787549 4657 hybrid_clock.cc:648] HybridClock initialized: now 1790534761787532 us; error 34 us; skew 500 ppm
I20260927 18:46:01.789307 4657 webserver.cc:460] Webserver started at http://127.18.235.130:40077/ using document root <none> and password file <none>
I20260927 18:46:01.789805 4657 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:01.789870 4657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:01.790040 4657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:01.791893 4657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "0fc8d8f2035c46ecbc702062cbded2ed"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "876a43b4a9a28bc06ac1d81c99c078f7"
server_key_iv: "3d9515eda093344f913706c56769d778"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.792398 4657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "0fc8d8f2035c46ecbc702062cbded2ed"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "876a43b4a9a28bc06ac1d81c99c078f7"
server_key_iv: "3d9515eda093344f913706c56769d778"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.795665 4657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:46:01.797839 4671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.798941 4657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:01.799041 4657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "0fc8d8f2035c46ecbc702062cbded2ed"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "876a43b4a9a28bc06ac1d81c99c078f7"
server_key_iv: "3d9515eda093344f913706c56769d778"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.799124 4657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:01.833699 4657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:01.834385 4657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:01.834583 4657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:01.835170 4657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:01.836107 4657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:01.836159 4657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.836216 4657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:01.836249 4657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.845156 4657 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:35885
I20260927 18:46:01.845245 4784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:35885 every 8 connection(s)
I20260927 18:46:01.846036 4657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:01.849000 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4657
I20260927 18:46:01.849130 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:01.849432 19374 external_mini_cluster.cc:1428] Setting key ad40699e8388a1ea40ebf236b3ea52dd
I20260927 18:46:01.851462 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38667
--builtin_ntp_servers=127.18.235.148:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:46:01.855849 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.847736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:45479 (local address 127.18.235.190:38667)
0927 18:46:01.847877 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.847883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.848946 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 18:46:01.850660 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.850664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.850709 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.850786 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.851739 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.852449 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.853965 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.854181 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.854254 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.855354 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.855357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.855358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.855369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.855388 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.855395 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.855448 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.855675 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.855736 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:46:01.856899 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.848132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38667 (local address 127.18.235.130:45479)
0927 18:46:01.848775 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.848828 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:01.850469 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:01.851000 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:01.851009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:01.851101 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:01.851610 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.851627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.852551 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.852553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.853843 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.853855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.854314 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.854318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.854496 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.855058 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:01.855060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:01.855115 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:01.855205 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:01.855473 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:01.855482 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:01.855487 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:46:01.855615 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:46:01.855903 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260927 18:46:01.858057 4785 heartbeater.cc:347] Connected to a master server at 127.18.235.190:38667
I20260927 18:46:01.858263 4785 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:01.858741 4785 heartbeater.cc:511] Master 127.18.235.190:38667 requested a full tablet report, sending...
I20260927 18:46:01.859690 4466 ts_manager.cc:194] Registered new tserver with Master: 0fc8d8f2035c46ecbc702062cbded2ed (127.18.235.130:35885)
I20260927 18:46:01.860219 4466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.130:45479
W20260927 18:46:01.955623 4788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:01.955842 4788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:01.955889 4788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:01.959781 4788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:01.959824 4788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:01.959893 4788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:46:01.964349 4788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:01.965234 4788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:01.966430 4788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:01.974916 4793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.974938 4794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.974967 4796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.975273 4788 server_base.cc:1034] running on GCE node
I20260927 18:46:01.975648 4788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:01.976239 4788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:01.977423 4788 hybrid_clock.cc:648] HybridClock initialized: now 1790534761977404 us; error 27 us; skew 500 ppm
I20260927 18:46:01.979136 4788 webserver.cc:460] Webserver started at http://127.18.235.131:45355/ using document root <none> and password file <none>
I20260927 18:46:01.979660 4788 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:01.979727 4788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:01.979907 4788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:01.981900 4788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "f921c100e899422883e00b5028e3352a"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "a47c27d834a6d85939c6eefe0e70ed54"
server_key_iv: "8b58f310c74e8391f82286a26b66751b"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.982438 4788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "f921c100e899422883e00b5028e3352a"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "a47c27d834a6d85939c6eefe0e70ed54"
server_key_iv: "8b58f310c74e8391f82286a26b66751b"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.986024 4788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:01.988246 4802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.989377 4788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:46:01.989480 4788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "f921c100e899422883e00b5028e3352a"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-6lr6"
server_key: "a47c27d834a6d85939c6eefe0e70ed54"
server_key_iv: "8b58f310c74e8391f82286a26b66751b"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.989593 4788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:02.002380 4788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:02.002949 4788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:02.003144 4788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:02.003711 4788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:02.004772 4788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:02.004832 4788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:02.004877 4788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:02.004910 4788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:02.013598 4788 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:36899
I20260927 18:46:02.013614 4915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:36899 every 8 connection(s)
I20260927 18:46:02.014420 4788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:02.017298 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4788
I20260927 18:46:02.017387 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:02.017612 19374 external_mini_cluster.cc:1428] Setting key 8e560df21e8cf27313ecc4d4245ac77e
I20260927 18:46:02.022864 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.015968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:35881 (local address 127.18.235.190:38667)
0927 18:46:02.016104 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.016107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.017005 (+ 898us) server_negotiation.cc:407] Connection header received
0927 18:46:02.018522 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.018525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.018578 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.018672 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:02.019719 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.020234 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.021351 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.021558 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.021627 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.022517 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:02.022520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:02.022522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:02.022533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:02.022551 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:02.022558 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:02.022604 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.022717 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.022763 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:46:02.023607 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.016267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:38667 (local address 127.18.235.131:35881)
0927 18:46:02.016864 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.016901 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:02.018313 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:02.018834 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:02.018843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:02.018949 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:02.019587 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.019604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.020326 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.020328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.021257 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.021267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.021674 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.021677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.021811 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.022291 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:02.022292 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:02.022328 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:02.022387 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:02.022617 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:02.022621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:02.022625 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:46:02.022713 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:46:02.022910 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
I20260927 18:46:02.024577 4916 heartbeater.cc:347] Connected to a master server at 127.18.235.190:38667
I20260927 18:46:02.024780 4916 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:02.025277 4916 heartbeater.cc:511] Master 127.18.235.190:38667 requested a full tablet report, sending...
I20260927 18:46:02.026175 4466 ts_manager.cc:194] Registered new tserver with Master: f921c100e899422883e00b5028e3352a (127.18.235.131:36899)
I20260927 18:46:02.026738 4466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.131:35881
I20260927 18:46:02.031035 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:02.036230 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.032762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.18.235.190:38667)
0927 18:46:02.032870 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.032874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.033061 (+ 187us) server_negotiation.cc:407] Connection header received
0927 18:46:02.033205 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.033208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.033249 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.033312 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:02.033580 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.034065 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.034870 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.035082 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.035165 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.035875 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:02.035879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:02.035882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:02.035893 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:02.035912 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:02.035920 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:02.035974 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.036060 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.036110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:02.038684 4466 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:02.041785 4466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:02.054582 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.049247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33939 (local address 127.0.0.1:43778)
0927 18:46:02.049560 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.049616 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:02.049780 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:02.050650 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:02.050666 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:02.050685 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:02.050796 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.050802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.051914 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.051917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.052735 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.052744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.054096 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.054099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.054210 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.054249 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:46:02.054346 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:46:02.054450 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260927 18:46:02.054649 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.049327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.18.235.129:33939)
0927 18:46:02.050009 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.050020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.050045 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:46:02.050124 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.050129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.050330 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.050486 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:02.050898 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.051814 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.053207 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.053995 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.054110 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.054277 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.054388 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.054484 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":109,"threads_started":1}
I20260927 18:46:02.055241 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.049565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:36899 (local address 127.0.0.1:47598)
0927 18:46:02.049800 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.049812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:02.050019 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:02.051313 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:02.051315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:02.051346 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:02.051446 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.051452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.052951 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.052954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.053864 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.053873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.054884 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.054888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.054973 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.054996 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:02.055050 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:46:02.055112 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"spinlock_wait_cycles":3840,"thread_start_us":92,"threads_started":1}
I20260927 18:46:02.055250 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.050294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.18.235.131:36899)
0927 18:46:02.050894 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.050905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.050922 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:46:02.050984 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.050988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.051161 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.051578 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:02.051711 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.052760 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.053978 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.054737 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.054886 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.055035 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.055073 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.055137 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":88,"threads_started":1}
I20260927 18:46:02.055670 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.049922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:35885 (local address 127.0.0.1:58962)
0927 18:46:02.050501 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.050513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:02.050581 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:02.051308 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:02.051311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:02.051344 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:02.051454 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.051460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.052808 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.052812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.053937 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.053949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.055307 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.055310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.055392 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.055414 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:02.055488 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:46:02.055549 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":18,"spinlock_wait_cycles":10496,"thread_start_us":88,"threads_started":1}
I20260927 18:46:02.055742 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.050450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.18.235.130:35885)
0927 18:46:02.050869 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.050881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.050905 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:46:02.050977 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.050983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.051169 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.051322 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:02.051552 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.052683 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.054073 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.054986 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.055093 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.055247 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.055532 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.055613 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":93,"threads_started":1}
I20260927 18:46:02.057967 4588 tablet_service.cc:1467] Processing CreateTablet for tablet 9dede8b947b34bc9a9f5de1bdf8242ae (DEFAULT_TABLE table=test-table [id=2f92df0d3a8a4ed292c19450d8197b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:02.058833 4588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dede8b947b34bc9a9f5de1bdf8242ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:02.058571 4719 tablet_service.cc:1467] Processing CreateTablet for tablet 9dede8b947b34bc9a9f5de1bdf8242ae (DEFAULT_TABLE table=test-table [id=2f92df0d3a8a4ed292c19450d8197b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:02.058568 4850 tablet_service.cc:1467] Processing CreateTablet for tablet 9dede8b947b34bc9a9f5de1bdf8242ae (DEFAULT_TABLE table=test-table [id=2f92df0d3a8a4ed292c19450d8197b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:02.059542 4850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dede8b947b34bc9a9f5de1bdf8242ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:02.059542 4719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dede8b947b34bc9a9f5de1bdf8242ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:02.065932 4933 tablet_bootstrap.cc:492] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195: Bootstrap starting.
I20260927 18:46:02.066077 4935 tablet_bootstrap.cc:492] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Bootstrap starting.
I20260927 18:46:02.066798 4934 tablet_bootstrap.cc:492] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Bootstrap starting.
I20260927 18:46:02.068130 4935 tablet_bootstrap.cc:654] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:02.068130 4933 tablet_bootstrap.cc:654] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:02.068370 4934 tablet_bootstrap.cc:654] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:02.068948 4933 log.cc:824] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:02.068948 4935 log.cc:824] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:02.068981 4934 log.cc:824] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:02.070597 4933 tablet_bootstrap.cc:492] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195: No bootstrap required, opened a new log
I20260927 18:46:02.070677 4934 tablet_bootstrap.cc:492] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: No bootstrap required, opened a new log
I20260927 18:46:02.070776 4933 ts_tablet_manager.cc:1397] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:02.070875 4934 ts_tablet_manager.cc:1397] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:02.070953 4935 tablet_bootstrap.cc:492] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: No bootstrap required, opened a new log
I20260927 18:46:02.071149 4935 ts_tablet_manager.cc:1397] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:02.073853 4933 raft_consensus.cc:348] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.074047 4933 raft_consensus.cc:374] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:02.074115 4933 raft_consensus.cc:729] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b8e83b72f64d8cb4d42f8b8d970195, State: Initialized, Role: FOLLOWER
I20260927 18:46:02.074502 4933 consensus_queue.cc:260] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.075068 4654 heartbeater.cc:503] Master 127.18.235.190:38667 was elected leader, sending a full tablet report...
I20260927 18:46:02.074921 4935 raft_consensus.cc:348] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.075152 4935 raft_consensus.cc:374] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:02.075234 4935 raft_consensus.cc:729] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f921c100e899422883e00b5028e3352a, State: Initialized, Role: FOLLOWER
I20260927 18:46:02.075373 4933 ts_tablet_manager.cc:1428] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:02.075608 4935 consensus_queue.cc:260] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.076160 4916 heartbeater.cc:503] Master 127.18.235.190:38667 was elected leader, sending a full tablet report...
I20260927 18:46:02.076486 4935 ts_tablet_manager.cc:1428] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:46:02.074921 4934 raft_consensus.cc:348] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.077028 4934 raft_consensus.cc:374] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:02.077131 4934 raft_consensus.cc:729] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc8d8f2035c46ecbc702062cbded2ed, State: Initialized, Role: FOLLOWER
I20260927 18:46:02.077541 4934 consensus_queue.cc:260] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.078372 4934 ts_tablet_manager.cc:1428] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:46:02.078687 4785 heartbeater.cc:503] Master 127.18.235.190:38667 was elected leader, sending a full tablet report...
I20260927 18:46:02.079614 4939 raft_consensus.cc:482] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:02.079748 4939 raft_consensus.cc:504] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.080780 4939 leader_election.cc:288] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fc8d8f2035c46ecbc702062cbded2ed (127.18.235.130:35885), f921c100e899422883e00b5028e3352a (127.18.235.131:36899)
I20260927 18:46:02.084265 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.081243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:56903 (local address 127.18.235.131:36899)
0927 18:46:02.081345 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.081349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.081362 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:02.081399 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.081402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.081438 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.081511 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:02.081812 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.082365 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.083400 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.083768 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.083866 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.083906 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.084095 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.084161 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:46:02.084270 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.081147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:33991 (local address 127.18.235.130:35885)
0927 18:46:02.081345 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.081349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.081362 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:02.081399 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.081402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.081438 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.081511 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:02.081818 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.082364 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.083395 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.083755 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.083842 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.083884 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.084107 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.084166 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 18:46:02.084283 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.080910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:35885 (local address 127.18.235.129:33991)
0927 18:46:02.081001 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.081015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:02.081083 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:02.081523 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:02.081526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:02.081560 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:02.081700 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.081708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.082463 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.082466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.083286 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.083295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.083858 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.083862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.083956 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.084011 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 18:46:02.084073 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:46:02.084141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":48,"spinlock_wait_cycles":23936}
I20260927 18:46:02.084300 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.080983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:36899 (local address 127.18.235.129:56903)
0927 18:46:02.081219 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.081229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:02.081285 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:02.081523 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:02.081525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:02.081558 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:02.081710 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.081716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.082471 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.082474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.083289 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:02.083298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.083883 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.083886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:02.083960 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.084008 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 18:46:02.084065 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:46:02.084127 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":79,"spinlock_wait_cycles":6272,"thread_start_us":86,"threads_started":1}
I20260927 18:46:02.084770 4739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dede8b947b34bc9a9f5de1bdf8242ae" candidate_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" is_pre_election: true
I20260927 18:46:02.084900 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dede8b947b34bc9a9f5de1bdf8242ae" candidate_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f921c100e899422883e00b5028e3352a" is_pre_election: true
I20260927 18:46:02.084990 4739 raft_consensus.cc:2454] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b8e83b72f64d8cb4d42f8b8d970195 in term 0.
I20260927 18:46:02.085098 4870 raft_consensus.cc:2454] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b8e83b72f64d8cb4d42f8b8d970195 in term 0.
I20260927 18:46:02.085457 4544 leader_election.cc:302] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0fc8d8f2035c46ecbc702062cbded2ed, 13b8e83b72f64d8cb4d42f8b8d970195, f921c100e899422883e00b5028e3352a; no voters:
I20260927 18:46:02.085683 4939 raft_consensus.cc:2781] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:02.085772 4939 raft_consensus.cc:482] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:02.085825 4939 raft_consensus.cc:3037] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:02.086671 4939 raft_consensus.cc:504] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.086913 4939 leader_election.cc:288] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [CANDIDATE]: Term 1 election: Requested vote from peers 0fc8d8f2035c46ecbc702062cbded2ed (127.18.235.130:35885), f921c100e899422883e00b5028e3352a (127.18.235.131:36899)
I20260927 18:46:02.087185 4739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dede8b947b34bc9a9f5de1bdf8242ae" candidate_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc8d8f2035c46ecbc702062cbded2ed"
I20260927 18:46:02.087199 4870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dede8b947b34bc9a9f5de1bdf8242ae" candidate_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f921c100e899422883e00b5028e3352a"
I20260927 18:46:02.087291 4739 raft_consensus.cc:3037] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:02.087291 4870 raft_consensus.cc:3037] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:02.088318 4870 raft_consensus.cc:2454] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b8e83b72f64d8cb4d42f8b8d970195 in term 1.
I20260927 18:46:02.088318 4739 raft_consensus.cc:2454] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b8e83b72f64d8cb4d42f8b8d970195 in term 1.
I20260927 18:46:02.088567 4544 leader_election.cc:302] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc8d8f2035c46ecbc702062cbded2ed, 13b8e83b72f64d8cb4d42f8b8d970195; no voters:
I20260927 18:46:02.088714 4939 raft_consensus.cc:2781] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:02.088894 4939 raft_consensus.cc:686] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [term 1 LEADER]: Becoming Leader. State: Replica: 13b8e83b72f64d8cb4d42f8b8d970195, State: Running, Role: LEADER
I20260927 18:46:02.089193 4939 consensus_queue.cc:237] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } }
I20260927 18:46:02.091625 4465 catalog_manager.cc:5495] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b8e83b72f64d8cb4d42f8b8d970195 (127.18.235.129). New cstate: current_term: 1 leader_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b8e83b72f64d8cb4d42f8b8d970195" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 33939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 } health_report { overall_health: UNKNOWN } } }
W20260927 18:46:02.096853 4786 tablet.cc:2369] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:02.102571 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.099781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.18.235.129:33939)
0927 18:46:02.099866 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.099869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.099894 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:46:02.100010 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.100012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.100053 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.100130 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:02.100288 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.100722 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.101491 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:02.101720 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:02.101791 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:02.102003 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:02.102332 (+ 329us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:02.102407 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.102455 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.102491 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:46:02.107200 4739 raft_consensus.cc:1261] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 1 FOLLOWER]: Refusing update from remote peer 13b8e83b72f64d8cb4d42f8b8d970195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:02.107208 4870 raft_consensus.cc:1261] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 1 FOLLOWER]: Refusing update from remote peer 13b8e83b72f64d8cb4d42f8b8d970195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:02.107745 4939 consensus_queue.cc:1035] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc8d8f2035c46ecbc702062cbded2ed" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:02.107908 4943 consensus_queue.cc:1035] T 9dede8b947b34bc9a9f5de1bdf8242ae P 13b8e83b72f64d8cb4d42f8b8d970195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f921c100e899422883e00b5028e3352a" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:02.117138 4947 mvcc.cc:205] Tried to move back new op lower bound from 7334030385585393664 to 7334030385519501312. Current Snapshot: MvccSnapshot[applied={T|T < 7334030385585393664 or (T in {7334030385585393664})}]
I20260927 18:46:02.121555 4466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:02.121672 4466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:02.123641 4466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8efa855551f64467ba2976c434e809d8: Sending DeleteTablet for 3 replicas of tablet 9dede8b947b34bc9a9f5de1bdf8242ae
I20260927 18:46:02.124292 4719 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dede8b947b34bc9a9f5de1bdf8242ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:02 UTC) from {username='slave'} at 127.0.0.1:58962
I20260927 18:46:02.124292 4588 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dede8b947b34bc9a9f5de1bdf8242ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:02 UTC) from {username='slave'} at 127.0.0.1:43778
I20260927 18:46:02.124332 4850 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dede8b947b34bc9a9f5de1bdf8242ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:02 UTC) from {username='slave'} at 127.0.0.1:47598
I20260927 18:46:02.124713 4957 tablet_replica.cc:330] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: stopping tablet replica
I20260927 18:46:02.124935 4955 tablet_replica.cc:330] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: stopping tablet replica
I20260927 18:46:02.124972 4957 raft_consensus.cc:2229] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:02.125212 4955 raft_consensus.cc:2229] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:02.125255 4957 raft_consensus.cc:2258] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:02.125372 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4526
I20260927 18:46:02.126968 4957 ts_tablet_manager.cc:1905] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:02.127487 4955 raft_consensus.cc:2258] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:02.128505 4955 ts_tablet_manager.cc:1905] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:02.129341 4957 ts_tablet_manager.cc:1918] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:02.129426 4957 log.cc:1196] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/9dede8b947b34bc9a9f5de1bdf8242ae
I20260927 18:46:02.129683 4957 ts_tablet_manager.cc:1939] T 9dede8b947b34bc9a9f5de1bdf8242ae P f921c100e899422883e00b5028e3352a: Deleting consensus metadata
I20260927 18:46:02.130503 4955 ts_tablet_manager.cc:1918] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:02.130576 4955 log.cc:1196] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/9dede8b947b34bc9a9f5de1bdf8242ae
I20260927 18:46:02.130698 4452 catalog_manager.cc:4841] TS f921c100e899422883e00b5028e3352a (127.18.235.131:36899): tablet 9dede8b947b34bc9a9f5de1bdf8242ae (table test-table [id=2f92df0d3a8a4ed292c19450d8197b84]) successfully deleted
I20260927 18:46:02.130834 4955 ts_tablet_manager.cc:1939] T 9dede8b947b34bc9a9f5de1bdf8242ae P 0fc8d8f2035c46ecbc702062cbded2ed: Deleting consensus metadata
W20260927 18:46:02.131742 4453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:33939 (error 108)
I20260927 18:46:02.131937 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4657
I20260927 18:46:02.132179 4453 catalog_manager.cc:4841] TS 0fc8d8f2035c46ecbc702062cbded2ed (127.18.235.130:35885): tablet 9dede8b947b34bc9a9f5de1bdf8242ae (table test-table [id=2f92df0d3a8a4ed292c19450d8197b84]) successfully deleted
I20260927 18:46:02.133311 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.132394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:33939 (local address 127.0.0.1:43796)
0927 18:46:02.133195 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:02.133253 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:33939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":727}
W20260927 18:46:02.133450 4453 catalog_manager.cc:4584] TS 13b8e83b72f64d8cb4d42f8b8d970195 (127.18.235.129:33939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dede8b947b34bc9a9f5de1bdf8242ae: Network error: Client connection negotiation failed: client connection to 127.18.235.129:33939: connect: Connection refused (error 111)
I20260927 18:46:02.136660 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4788
I20260927 18:46:02.141088 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4435
2026-09-27T18:46:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (894 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:02 dist-test-slave-6lr6 krb5kdc[4961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:02 dist-test-slave-6lr6 krb5kdc[4961](info): set up 2 sockets
Sep 27 18:46:02 dist-test-slave-6lr6 krb5kdc[4961](info): commencing operation
krb5kdc: starting...
W20260927 18:46:03.081055 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:03.191833 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:34389
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:03.293810 4974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.294027 4974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.294088 4974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.297920 4974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:03.297966 4974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.297992 4974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.298012 4974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:03.298038 4974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:03.302899 4974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:34389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:03.303726 4974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.304570 4974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.312047 4980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.312038 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.312198 4974 server_base.cc:1034] running on GCE node
W20260927 18:46:03.312036 4979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.313068 4974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.314023 4974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.315179 4974 hybrid_clock.cc:648] HybridClock initialized: now 1790534763315165 us; error 25 us; skew 500 ppm
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:03.317623 4974 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:03.318401 4974 webserver.cc:460] Webserver started at http://127.18.235.190:38017/ using document root <none> and password file <none>
I20260927 18:46:03.318969 4974 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.319044 4974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.319224 4974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.321142 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "0921c5a15586467eada3d7247f248847"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "4a2d7ce2761e609a3dbf16b6cae6d397"
server_key_iv: "6d79a04885e4fa52e6a583b6f9fb98e5"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.321669 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "0921c5a15586467eada3d7247f248847"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "4a2d7ce2761e609a3dbf16b6cae6d397"
server_key_iv: "6d79a04885e4fa52e6a583b6f9fb98e5"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.325212 4974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:03.327438 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.328689 4974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:03.328806 4974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "0921c5a15586467eada3d7247f248847"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "4a2d7ce2761e609a3dbf16b6cae6d397"
server_key_iv: "6d79a04885e4fa52e6a583b6f9fb98e5"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.328896 4974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:03.340423 4974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:03.342973 4974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:03.343124 4974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:03.350034 4974 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:34389
I20260927 18:46:03.350044 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:34389 every 8 connection(s)
I20260927 18:46:03.350875 4974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:03.353457 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:03.356268 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 4974
I20260927 18:46:03.356426 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:03.356739 19374 external_mini_cluster.cc:1428] Setting key 600756c85c344ab017953c9ce0ccf9bd
I20260927 18:46:03.359122 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: Bootstrap starting.
I20260927 18:46:03.361649 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:03.362468 5042 log.cc:824] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:03.364112 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: No bootstrap required, opened a new log
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:03.367651 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0921c5a15586467eada3d7247f248847" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 34389 } }
I20260927 18:46:03.367852 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:03.367964 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0921c5a15586467eada3d7247f248847, State: Initialized, Role: FOLLOWER
I20260927 18:46:03.368384 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0921c5a15586467eada3d7247f248847" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 34389 } }
I20260927 18:46:03.368517 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:03.368594 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:03.368697 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:03.369572 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0921c5a15586467eada3d7247f248847" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 34389 } }
I20260927 18:46:03.369894 5042 leader_election.cc:302] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0921c5a15586467eada3d7247f248847; no voters:
I20260927 18:46:03.370158 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:03.370240 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:03.370446 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [term 1 LEADER]: Becoming Leader. State: Replica: 0921c5a15586467eada3d7247f248847, State: Running, Role: LEADER
I20260927 18:46:03.370575 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.358003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.18.235.190:34389)
0927 18:46:03.358376 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:03.358390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:03.358422 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:46:03.359034 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:03.359051 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:03.359380 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:03.359631 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:03.360714 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.361864 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.363020 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.363469 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.363627 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:03.366156 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:03.366161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:03.366172 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:03.366195 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:03.367822 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:03.368250 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:03.368254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:03.368258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:03.368361 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:03.368616 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:03.368619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:03.368620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:03.369009 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:03.369218 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:03.369476 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:46:03.369751 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260927 18:46:03.370805 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0921c5a15586467eada3d7247f248847" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 34389 } }
I20260927 18:46:03.371241 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:03.373730 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0921c5a15586467eada3d7247f248847. Latest consensus state: current_term: 1 leader_uuid: "0921c5a15586467eada3d7247f248847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0921c5a15586467eada3d7247f248847" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 34389 } } }
I20260927 18:46:03.373829 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:03.374032 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0921c5a15586467eada3d7247f248847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0921c5a15586467eada3d7247f248847" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 34389 } } }
I20260927 18:46:03.374096 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847 [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:03.376088 5062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:03.376148 5062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:03.376230 5063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:03.377220 5063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:03.382058 5063 catalog_manager.cc:1347] Generated new cluster ID: 2dae52ba3f0b46569e2545ce8ca8f30a
I20260927 18:46:03.382128 5063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:03.410516 5063 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:03.411729 5063 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:03.417009 5063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: Generated new TSK 0
I20260927 18:46:03.417770 5063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:46:03.449936 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:34389
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:03.553473 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.553704 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.553766 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.557615 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.557669 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.557747 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:46:03.562733 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:03.563585 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.564464 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.573158 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.573329 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.573362 5068 server_base.cc:1034] running on GCE node
W20260927 18:46:03.573380 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.573777 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.574353 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.575526 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790534763575510 us; error 29 us; skew 500 ppm
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:03.578025 5068 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:46:03.578792 5068 webserver.cc:460] Webserver started at http://127.18.235.129:37679/ using document root <none> and password file <none>
I20260927 18:46:03.579264 5068 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.579325 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.579488 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.581357 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "a31035fbcff94442bb7fff00721d69d8"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "276b02afa99f120372de686231a2ed8e"
server_key_iv: "6a8709baf479177151467f7d62569daf"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.581857 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "a31035fbcff94442bb7fff00721d69d8"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "276b02afa99f120372de686231a2ed8e"
server_key_iv: "6a8709baf479177151467f7d62569daf"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.585166 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 18:46:03.587283 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.588348 5068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 18:46:03.588444 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "a31035fbcff94442bb7fff00721d69d8"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "276b02afa99f120372de686231a2ed8e"
server_key_iv: "6a8709baf479177151467f7d62569daf"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.588527 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:03.605247 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:03.607940 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:03.608093 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:03.608573 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:03.609297 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:03.609346 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.609382 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:03.609411 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.617090 5068 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:36339
I20260927 18:46:03.617159 5196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:36339 every 8 connection(s)
I20260927 18:46:03.617868 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:03.624886 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5068
I20260927 18:46:03.624995 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:03.625290 19374 external_mini_cluster.cc:1428] Setting key 0d41288583b5382958f442481b88c7a4
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:03.630270 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.619517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:33263 (local address 127.18.235.190:34389)
0927 18:46:03.619660 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:03.619663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:03.620394 (+ 731us) server_negotiation.cc:407] Connection header received
0927 18:46:03.621664 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:03.621667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:03.621712 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:03.621783 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:03.622775 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.623297 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.624345 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.624580 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.624655 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:03.627130 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:03.627133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:03.627135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:03.627157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:03.628412 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:03.628855 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:03.628859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:03.628860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:03.628934 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:03.629304 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:03.629307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:03.629308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:03.629478 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:03.629555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:03.630054 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 18:46:03.630149 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:03.631258 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.619867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:34389 (local address 127.18.235.129:33263)
0927 18:46:03.620257 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:03.620296 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:03.621471 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:03.621915 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:03.621921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:03.622215 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:03.622657 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:03.622671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.623394 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.623396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:03.624242 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:03.624250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.624739 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.624743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:03.624879 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:03.625426 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:03.625441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:03.626979 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:03.628515 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:03.628520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:03.628533 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:03.628758 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:03.629016 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:03.629019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:03.629021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:03.629206 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:03.629569 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:03.629575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:03.629837 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 18:46:03.630007 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 18:46:03.630318 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":112,"threads_started":1}
I20260927 18:46:03.632366 5197 heartbeater.cc:347] Connected to a master server at 127.18.235.190:34389
I20260927 18:46:03.632635 5197 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:03.633186 5197 heartbeater.cc:511] Master 127.18.235.190:34389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:46:03.634568 5006 ts_manager.cc:194] Registered new tserver with Master: a31035fbcff94442bb7fff00721d69d8 (127.18.235.129:36339)
I20260927 18:46:03.635588 5006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:33263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:46:03.649345 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:34389
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:03.748315 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.748513 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.748575 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.752372 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.752416 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.752490 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:46:03.757308 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:03.758096 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.758917 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.767143 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.767246 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.767282 5202 server_base.cc:1034] running on GCE node
W20260927 18:46:03.767148 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.767709 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.768255 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.769405 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790534763769384 us; error 32 us; skew 500 ppm
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:03.772012 5202 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:46:03.772877 5202 webserver.cc:460] Webserver started at http://127.18.235.130:35803/ using document root <none> and password file <none>
I20260927 18:46:03.773413 5202 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.773476 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.773661 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.775457 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "ef0ce174afd940559ee84df83aa435e3"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "7f9d4c5a7c72fd502a9999ce0b24ec82"
server_key_iv: "c91eecf4a76cae792b8acdb4695bca1f"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.775960 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "ef0ce174afd940559ee84df83aa435e3"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "7f9d4c5a7c72fd502a9999ce0b24ec82"
server_key_iv: "c91eecf4a76cae792b8acdb4695bca1f"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.779342 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:03.781531 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.782559 5202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:03.782671 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "ef0ce174afd940559ee84df83aa435e3"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "7f9d4c5a7c72fd502a9999ce0b24ec82"
server_key_iv: "c91eecf4a76cae792b8acdb4695bca1f"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.782758 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:03.796303 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:03.798945 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:03.799093 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:03.799563 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:03.800242 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:03.800288 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.800334 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:03.800360 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.807876 5202 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:42695
I20260927 18:46:03.807904 5330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:42695 every 8 connection(s)
I20260927 18:46:03.808574 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:03.813665 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5202
I20260927 18:46:03.813792 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:03.814134 19374 external_mini_cluster.cc:1428] Setting key 55b766705658d77a00b3b3e4210ec6a8
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:03.821165 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.810147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:50109 (local address 127.18.235.190:34389)
0927 18:46:03.810305 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:03.810309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:03.810986 (+ 677us) server_negotiation.cc:407] Connection header received
0927 18:46:03.812136 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:03.812139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:03.812181 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:03.812253 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:03.813268 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.813770 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.815196 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.815428 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.815514 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:03.817766 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:03.817770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:03.817773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:03.817799 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:03.819227 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:03.819726 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:03.819729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:03.819731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:03.819832 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:03.820120 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:03.820124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:03.820126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:03.820336 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:03.820436 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:03.820859 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:46:03.820975 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:46:03.821791 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.810437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:34389 (local address 127.18.235.130:50109)
0927 18:46:03.810852 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:03.810891 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:03.811968 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:03.812380 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:03.812386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:03.812716 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:03.813152 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:03.813165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.813896 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.813900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:03.815078 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:03.815088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:03.815556 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:03.815560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:03.815705 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:03.816145 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:03.816160 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:03.817593 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:03.819351 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:03.819355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:03.819366 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:03.819632 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:03.819917 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:03.819919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:03.819920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:03.820039 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:03.820423 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:03.820427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:03.820649 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 18:46:03.820795 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 18:46:03.821011 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:46:03.822784 5331 heartbeater.cc:347] Connected to a master server at 127.18.235.190:34389
I20260927 18:46:03.822988 5331 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:03.823385 5331 heartbeater.cc:511] Master 127.18.235.190:34389 requested a full tablet report, sending...
I20260927 18:46:03.824276 5006 ts_manager.cc:194] Registered new tserver with Master: ef0ce174afd940559ee84df83aa435e3 (127.18.235.130:42695)
I20260927 18:46:03.824915 5006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:50109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:46:03.838038 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:34389
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:03.938138 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.938360 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.938400 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.942325 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.942369 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.942428 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:46:03.947296 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:03.948108 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.948954 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.957284 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.957322 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.957320 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.957566 5336 server_base.cc:1034] running on GCE node
I20260927 18:46:03.957880 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.958387 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.959576 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790534763959560 us; error 29 us; skew 500 ppm
Sep 27 18:46:03 dist-test-slave-6lr6 krb5kdc[4961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:03.961963 5336 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:46:03.962723 5336 webserver.cc:460] Webserver started at http://127.18.235.131:37055/ using document root <none> and password file <none>
I20260927 18:46:03.963182 5336 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.963239 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.963418 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.965323 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "7f5d484aeefc40f39afaff3405c1371f"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "1d9e4768425d4075236ea0b3d27134d3"
server_key_iv: "cde0f56955090c85369f24d1a9d45be5"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.965801 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "7f5d484aeefc40f39afaff3405c1371f"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "1d9e4768425d4075236ea0b3d27134d3"
server_key_iv: "cde0f56955090c85369f24d1a9d45be5"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.968921 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:46:03.970955 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.972080 5336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:03.972174 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "7f5d484aeefc40f39afaff3405c1371f"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-6lr6"
server_key: "1d9e4768425d4075236ea0b3d27134d3"
server_key_iv: "cde0f56955090c85369f24d1a9d45be5"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.972265 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:04.008764 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:04.011592 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:04.011760 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:04.012291 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:04.013053 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:04.013118 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.013156 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:04.013176 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.021023 5336 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:45033
I20260927 18:46:04.021052 5464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:45033 every 8 connection(s)
I20260927 18:46:04.021811 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:04.022635 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5336
I20260927 18:46:04.022735 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:04.022979 19374 external_mini_cluster.cc:1428] Setting key 37b46d4268776a5f09448a99f85b1ef9
Sep 27 18:46:04 dist-test-slave-6lr6 krb5kdc[4961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:04.033491 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.023384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:41109 (local address 127.18.235.190:34389)
0927 18:46:04.023520 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.023525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.024315 (+ 790us) server_negotiation.cc:407] Connection header received
0927 18:46:04.025652 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.025655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.025705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.025785 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.026821 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.027330 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.028407 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.028632 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.028708 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.030719 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.030722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.030724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:04.030748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.031837 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:04.032245 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:04.032247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:04.032249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:04.032313 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:04.032540 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:04.032542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:04.032543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:04.032689 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.032755 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.033264 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.033369 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:46:04.034134 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.023780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:34389 (local address 127.18.235.131:41109)
0927 18:46:04.024187 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.024224 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.025469 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.025949 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.025955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.026273 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:04.026708 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.026721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.027452 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.027455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.028303 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.028312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.028736 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.028738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.028862 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.029294 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:04.029309 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:04.030584 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:04.031934 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:04.031938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:04.031948 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:04.032176 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:04.032371 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:04.032373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:04.032374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:04.032483 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:04.032762 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:04.032766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:04.033012 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.033196 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.033407 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
I20260927 18:46:04.035104 5465 heartbeater.cc:347] Connected to a master server at 127.18.235.190:34389
I20260927 18:46:04.035288 5465 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:04.035683 5465 heartbeater.cc:511] Master 127.18.235.190:34389 requested a full tablet report, sending...
I20260927 18:46:04.036615 5005 ts_manager.cc:194] Registered new tserver with Master: 7f5d484aeefc40f39afaff3405c1371f (127.18.235.131:45033)
I20260927 18:46:04.037288 5005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:41109
I20260927 18:46:04.037412 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:04.049289 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.039408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.18.235.190:34389)
0927 18:46:04.039490 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.039494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.039698 (+ 204us) server_negotiation.cc:407] Connection header received
0927 18:46:04.039794 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.039797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.039835 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.039899 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.040514 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.040983 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.041785 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.041997 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.042086 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.046745 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.046749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.046751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:04.046771 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.047734 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:04.048172 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:04.048175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:04.048176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:04.048239 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:04.048523 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:04.048525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:04.048526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:04.048694 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.048817 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.049053 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.049179 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 18:46:04.051623 5005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:04.054351 5005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:04.067375 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.062224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.18.235.129:36339)
0927 18:46:04.062485 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.062494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.062516 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:46:04.062587 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.062591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.062800 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.062943 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:04.063319 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.064372 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.066093 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.066845 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.066946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.067039 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.067159 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.067251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260927 18:46:04.067394 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.062125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:45033 (local address 127.0.0.1:47042)
0927 18:46:04.062873 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.062884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.062962 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.063468 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.063471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.063486 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:04.063561 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.063566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.064740 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.064743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.065876 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.065885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.066859 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.066861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.066964 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.067002 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.067115 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.067252 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":20,"thread_start_us":83,"threads_started":1}
I20260927 18:46:04.067394 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.061655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:36339 (local address 127.0.0.1:50704)
0927 18:46:04.061997 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.062010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.062120 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.063098 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.063102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.063121 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:04.063200 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.063205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.064490 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.064493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.065972 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.065982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.066943 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.066946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.067018 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.067035 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.067115 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.067262 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":89,"thread_start_us":96,"threads_started":1}
I20260927 18:46:04.067749 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.062409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.18.235.131:45033)
0927 18:46:04.062659 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.062668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.063037 (+ 369us) server_negotiation.cc:407] Connection header received
0927 18:46:04.063097 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.063101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.063249 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.063359 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:04.063680 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.064635 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.065978 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.066759 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.066879 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.066994 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.067547 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.067629 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260927 18:46:04.068377 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.061505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:42695 (local address 127.0.0.1:41436)
0927 18:46:04.061957 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.061997 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.062119 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.063258 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.063261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.063276 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:04.063354 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.063360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.064358 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.064362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.065981 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.065987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.068008 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.068012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.068105 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.068127 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.068193 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.068255 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":67,"thread_start_us":74,"threads_started":1}
I20260927 18:46:04.068444 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.061642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.18.235.130:42695)
0927 18:46:04.062838 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.062849 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.062870 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:46:04.062942 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.062947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.063134 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.063271 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:04.063467 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.064236 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.067093 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.067893 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.067999 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.068079 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.068230 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.068314 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":105,"threads_started":1}
I20260927 18:46:04.070571 5399 tablet_service.cc:1467] Processing CreateTablet for tablet c9b036cd62f3438f82924e8837487433 (DEFAULT_TABLE table=test-table [id=712918aadade48689e80cb6f1d29e3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:04.070830 5131 tablet_service.cc:1467] Processing CreateTablet for tablet c9b036cd62f3438f82924e8837487433 (DEFAULT_TABLE table=test-table [id=712918aadade48689e80cb6f1d29e3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:04.071025 5265 tablet_service.cc:1467] Processing CreateTablet for tablet c9b036cd62f3438f82924e8837487433 (DEFAULT_TABLE table=test-table [id=712918aadade48689e80cb6f1d29e3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:04.071359 5399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b036cd62f3438f82924e8837487433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:04.071821 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b036cd62f3438f82924e8837487433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:04.071821 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b036cd62f3438f82924e8837487433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:04.077037 5482 tablet_bootstrap.cc:492] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Bootstrap starting.
I20260927 18:46:04.078989 5482 tablet_bootstrap.cc:654] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:04.078972 5484 tablet_bootstrap.cc:492] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Bootstrap starting.
I20260927 18:46:04.079406 5483 tablet_bootstrap.cc:492] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: Bootstrap starting.
I20260927 18:46:04.080061 5482 log.cc:824] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:04.080945 5484 tablet_bootstrap.cc:654] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:04.081316 5483 tablet_bootstrap.cc:654] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:04.081647 5482 tablet_bootstrap.cc:492] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: No bootstrap required, opened a new log
I20260927 18:46:04.081804 5482 ts_tablet_manager.cc:1397] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 18:46:04.082177 5484 log.cc:824] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:04.082405 5483 log.cc:824] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:04.083997 5483 tablet_bootstrap.cc:492] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: No bootstrap required, opened a new log
I20260927 18:46:04.084158 5483 ts_tablet_manager.cc:1397] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:04.084491 5484 tablet_bootstrap.cc:492] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: No bootstrap required, opened a new log
I20260927 18:46:04.084657 5484 ts_tablet_manager.cc:1397] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:46:04.085062 5482 raft_consensus.cc:348] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.085342 5482 raft_consensus.cc:374] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:04.085425 5482 raft_consensus.cc:729] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5d484aeefc40f39afaff3405c1371f, State: Initialized, Role: FOLLOWER
I20260927 18:46:04.085882 5482 consensus_queue.cc:260] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.086486 5465 heartbeater.cc:503] Master 127.18.235.190:34389 was elected leader, sending a full tablet report...
I20260927 18:46:04.086970 5482 ts_tablet_manager.cc:1428] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 18:46:04.087379 5483 raft_consensus.cc:348] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.087613 5483 raft_consensus.cc:374] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:04.087689 5483 raft_consensus.cc:729] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31035fbcff94442bb7fff00721d69d8, State: Initialized, Role: FOLLOWER
I20260927 18:46:04.088168 5483 consensus_queue.cc:260] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.088163 5484 raft_consensus.cc:348] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.088423 5484 raft_consensus.cc:374] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:04.088511 5484 raft_consensus.cc:729] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0ce174afd940559ee84df83aa435e3, State: Initialized, Role: FOLLOWER
I20260927 18:46:04.088862 5484 consensus_queue.cc:260] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.089093 5483 ts_tablet_manager.cc:1428] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:04.089449 5197 heartbeater.cc:503] Master 127.18.235.190:34389 was elected leader, sending a full tablet report...
I20260927 18:46:04.089738 5484 ts_tablet_manager.cc:1428] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:04.090047 5331 heartbeater.cc:503] Master 127.18.235.190:34389 was elected leader, sending a full tablet report...
W20260927 18:46:04.118850 5198 tablet.cc:2369] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:04.272645 5466 tablet.cc:2369] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:04.309547 5332 tablet.cc:2369] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:04.324622 5490 raft_consensus.cc:482] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:04.324798 5490 raft_consensus.cc:504] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.325872 5490 leader_election.cc:288] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a31035fbcff94442bb7fff00721d69d8 (127.18.235.129:36339), 7f5d484aeefc40f39afaff3405c1371f (127.18.235.131:45033)
I20260927 18:46:04.330823 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.326150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:57563 (local address 127.18.235.129:36339)
0927 18:46:04.326270 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.326274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.326452 (+ 178us) server_negotiation.cc:407] Connection header received
0927 18:46:04.326586 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.326588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.326649 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.326721 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:04.327108 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.327858 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.329615 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.330227 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.330343 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.330392 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.330636 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.330702 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:04.330823 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.326190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:57663 (local address 127.18.235.131:45033)
0927 18:46:04.326312 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.326317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.326481 (+ 164us) server_negotiation.cc:407] Connection header received
0927 18:46:04.326585 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.326588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.326639 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.326723 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:04.327088 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.327876 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.329626 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.330227 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.330343 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.330392 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.330630 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.330698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:46:04.330863 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.326049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:36339 (local address 127.18.235.130:57563)
0927 18:46:04.326354 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.326367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.326489 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.326737 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.326740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.326776 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:04.326997 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.327003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.327955 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.327959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.329495 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.329507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.330333 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.330336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.330473 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.330534 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.330602 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.330690 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":247,"spinlock_wait_cycles":6656,"thread_start_us":81,"threads_started":1}
I20260927 18:46:04.330870 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.326110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:45033 (local address 127.18.235.130:57663)
0927 18:46:04.326408 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.326419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.326497 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.326737 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.326740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.326774 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:04.326986 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.326992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.327956 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.327959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.329495 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.329507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.330333 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.330336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.330469 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.330532 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.330606 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.330690 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":225,"spinlock_wait_cycles":31872,"thread_start_us":71,"threads_started":1}
I20260927 18:46:04.331450 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b036cd62f3438f82924e8837487433" candidate_uuid: "ef0ce174afd940559ee84df83aa435e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31035fbcff94442bb7fff00721d69d8" is_pre_election: true
I20260927 18:46:04.331764 5151 raft_consensus.cc:2454] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0ce174afd940559ee84df83aa435e3 in term 0.
I20260927 18:46:04.331772 5419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b036cd62f3438f82924e8837487433" candidate_uuid: "ef0ce174afd940559ee84df83aa435e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5d484aeefc40f39afaff3405c1371f" is_pre_election: true
I20260927 18:46:04.332010 5419 raft_consensus.cc:2454] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0ce174afd940559ee84df83aa435e3 in term 0.
I20260927 18:46:04.332262 5219 leader_election.cc:302] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a31035fbcff94442bb7fff00721d69d8, ef0ce174afd940559ee84df83aa435e3; no voters:
I20260927 18:46:04.332535 5490 raft_consensus.cc:2781] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:04.332618 5490 raft_consensus.cc:482] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:04.332664 5490 raft_consensus.cc:3037] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:04.333782 5490 raft_consensus.cc:504] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.334074 5490 leader_election.cc:288] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [CANDIDATE]: Term 1 election: Requested vote from peers a31035fbcff94442bb7fff00721d69d8 (127.18.235.129:36339), 7f5d484aeefc40f39afaff3405c1371f (127.18.235.131:45033)
I20260927 18:46:04.334357 5419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b036cd62f3438f82924e8837487433" candidate_uuid: "ef0ce174afd940559ee84df83aa435e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5d484aeefc40f39afaff3405c1371f"
I20260927 18:46:04.334465 5419 raft_consensus.cc:3037] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:04.335466 5419 raft_consensus.cc:2454] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0ce174afd940559ee84df83aa435e3 in term 1.
I20260927 18:46:04.335736 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b036cd62f3438f82924e8837487433" candidate_uuid: "ef0ce174afd940559ee84df83aa435e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31035fbcff94442bb7fff00721d69d8"
I20260927 18:46:04.335785 5220 leader_election.cc:302] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f5d484aeefc40f39afaff3405c1371f, ef0ce174afd940559ee84df83aa435e3; no voters:
I20260927 18:46:04.335863 5151 raft_consensus.cc:3037] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:04.335927 5490 raft_consensus.cc:2781] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:04.336134 5490 raft_consensus.cc:686] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 1 LEADER]: Becoming Leader. State: Replica: ef0ce174afd940559ee84df83aa435e3, State: Running, Role: LEADER
I20260927 18:46:04.336444 5490 consensus_queue.cc:237] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } }
I20260927 18:46:04.336941 5151 raft_consensus.cc:2454] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0ce174afd940559ee84df83aa435e3 in term 1.
I20260927 18:46:04.339250 5004 catalog_manager.cc:5495] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0ce174afd940559ee84df83aa435e3 (127.18.235.130). New cstate: current_term: 1 leader_uuid: "ef0ce174afd940559ee84df83aa435e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0ce174afd940559ee84df83aa435e3" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 42695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:04.359848 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.356527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.18.235.130:42695)
0927 18:46:04.356637 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.356641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.356656 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:04.356808 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.356811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.356860 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.356941 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:04.357173 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.357689 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.358500 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.358714 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.358796 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.359122 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:04.359468 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:04.359589 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.359706 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.359740 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:46:04.365260 5151 raft_consensus.cc:1261] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 1 FOLLOWER]: Refusing update from remote peer ef0ce174afd940559ee84df83aa435e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:04.365262 5419 raft_consensus.cc:1261] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 1 FOLLOWER]: Refusing update from remote peer ef0ce174afd940559ee84df83aa435e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:04.365859 5490 consensus_queue.cc:1035] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5d484aeefc40f39afaff3405c1371f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:04.366025 5493 consensus_queue.cc:1035] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a31035fbcff94442bb7fff00721d69d8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:04.371194 5496 mvcc.cc:205] Tried to move back new op lower bound from 7334030394833678336 to 7334030394724462592. Current Snapshot: MvccSnapshot[applied={T|T < 7334030394833678336}]
I20260927 18:46:04.372563 5497 mvcc.cc:205] Tried to move back new op lower bound from 7334030394833678336 to 7334030394724462592. Current Snapshot: MvccSnapshot[applied={T|T < 7334030394833678336}]
I20260927 18:46:04.376174 5499 mvcc.cc:205] Tried to move back new op lower bound from 7334030394833678336 to 7334030394724462592. Current Snapshot: MvccSnapshot[applied={T|T < 7334030394833678336}]
I20260927 18:46:04.381345 5005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:04.381497 5005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:04.383615 5005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0921c5a15586467eada3d7247f248847: Sending DeleteTablet for 3 replicas of tablet c9b036cd62f3438f82924e8837487433
I20260927 18:46:04.384284 5265 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b036cd62f3438f82924e8837487433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:04 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:41436
I20260927 18:46:04.384284 5399 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b036cd62f3438f82924e8837487433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:04 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:47042
I20260927 18:46:04.384475 5131 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b036cd62f3438f82924e8837487433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:04 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:50704
I20260927 18:46:04.384686 5507 tablet_replica.cc:330] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: stopping tablet replica
I20260927 18:46:04.384943 5507 raft_consensus.cc:2229] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:04.385053 5508 tablet_replica.cc:330] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8: stopping tablet replica
I20260927 18:46:04.385236 5507 raft_consensus.cc:2258] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:04.385335 5508 raft_consensus.cc:2229] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:04.385547 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5068
I20260927 18:46:04.385591 5508 raft_consensus.cc:2258] T c9b036cd62f3438f82924e8837487433 P a31035fbcff94442bb7fff00721d69d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:04.386586 5507 ts_tablet_manager.cc:1905] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:04.388451 5506 tablet_replica.cc:330] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: stopping tablet replica
I20260927 18:46:04.388718 5506 raft_consensus.cc:2229] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:04.389045 5506 raft_consensus.cc:2258] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:04.389670 5507 ts_tablet_manager.cc:1918] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:04.389803 5507 log.cc:1196] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/c9b036cd62f3438f82924e8837487433
I20260927 18:46:04.390245 5507 ts_tablet_manager.cc:1939] T c9b036cd62f3438f82924e8837487433 P 7f5d484aeefc40f39afaff3405c1371f: Deleting consensus metadata
I20260927 18:46:04.391007 5506 ts_tablet_manager.cc:1905] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:04.391983 4992 catalog_manager.cc:4841] TS 7f5d484aeefc40f39afaff3405c1371f (127.18.235.131:45033): tablet c9b036cd62f3438f82924e8837487433 (table test-table [id=712918aadade48689e80cb6f1d29e3ae]) successfully deleted
I20260927 18:46:04.394553 5506 ts_tablet_manager.cc:1918] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:04.394642 5506 log.cc:1196] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/c9b036cd62f3438f82924e8837487433
I20260927 18:46:04.394914 5506 ts_tablet_manager.cc:1939] T c9b036cd62f3438f82924e8837487433 P ef0ce174afd940559ee84df83aa435e3: Deleting consensus metadata
W20260927 18:46:04.395282 4991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:36339 (error 108)
I20260927 18:46:04.396060 4992 catalog_manager.cc:4841] TS ef0ce174afd940559ee84df83aa435e3 (127.18.235.130:42695): tablet c9b036cd62f3438f82924e8837487433 (table test-table [id=712918aadade48689e80cb6f1d29e3ae]) successfully deleted
I20260927 18:46:04.396179 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5202
I20260927 18:46:04.397392 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.397140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:36339 (local address 127.0.0.1:50710)
0927 18:46:04.397250 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.397316 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 18:46:04.397629 4991 catalog_manager.cc:4584] TS a31035fbcff94442bb7fff00721d69d8 (127.18.235.129:36339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9b036cd62f3438f82924e8837487433: Network error: Client connection negotiation failed: client connection to 127.18.235.129:36339: connect: Connection refused (error 111)
I20260927 18:46:04.401292 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5336
I20260927 18:46:04.405637 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 4974
2026-09-27T18:46:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2265 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T18:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:04Z Disabled control of system clock
I20260927 18:46:04.432266 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:41737
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:41737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:46:04.534009 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:04.534232 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:04.534264 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:04.538132 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:04.538183 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:04.538197 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:04.538214 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:04.538230 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:04.542666 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:41737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:41737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:04.543457 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:04.544574 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:04.552022 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:04.552032 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:04.552032 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:04.552111 5513 server_base.cc:1034] running on GCE node
I20260927 18:46:04.552723 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:04.553720 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:04.554865 5513 hybrid_clock.cc:648] HybridClock initialized: now 1790534764554844 us; error 33 us; skew 500 ppm
I20260927 18:46:04.556493 5513 webserver.cc:460] Webserver started at http://127.18.235.190:42113/ using document root <none> and password file <none>
I20260927 18:46:04.556977 5513 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:04.557031 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:04.557248 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:04.559049 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "6bd7cecd826a435ba51b73528ea59ac2"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "8c43076e79d6e5b402775a89c05855e2"
server_key_iv: "8dd0245c7fb520d62acc5a6d1fca5a18"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.559549 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "6bd7cecd826a435ba51b73528ea59ac2"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "8c43076e79d6e5b402775a89c05855e2"
server_key_iv: "8dd0245c7fb520d62acc5a6d1fca5a18"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.562870 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:04.565093 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.566246 5513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:04.566351 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "6bd7cecd826a435ba51b73528ea59ac2"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "8c43076e79d6e5b402775a89c05855e2"
server_key_iv: "8dd0245c7fb520d62acc5a6d1fca5a18"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.566442 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:04.580911 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:04.581501 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:04.581664 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:04.588099 5513 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:41737
I20260927 18:46:04.588104 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:41737 every 8 connection(s)
I20260927 18:46:04.588936 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:04.591645 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:04.596519 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2: Bootstrap starting.
I20260927 18:46:04.596735 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5513
I20260927 18:46:04.596865 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:04.597157 19374 external_mini_cluster.cc:1428] Setting key a6692d4453fccf9e285d70a3ea727fc8
I20260927 18:46:04.598783 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:04.599436 5580 log.cc:824] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:04.601274 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2: No bootstrap required, opened a new log
I20260927 18:46:04.605010 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd7cecd826a435ba51b73528ea59ac2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41737 } }
I20260927 18:46:04.605234 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:04.605291 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd7cecd826a435ba51b73528ea59ac2, State: Initialized, Role: FOLLOWER
I20260927 18:46:04.605676 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.598312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.18.235.190:41737)
0927 18:46:04.598851 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.598867 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.598910 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:46:04.599501 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.599517 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.599950 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.600274 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.600580 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.601563 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.602725 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.603180 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.603295 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.604071 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.604076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.604088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:04.604425 (+ 337us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.604478 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:04.604499 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.604581 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.604697 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.604931 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260927 18:46:04.605696 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd7cecd826a435ba51b73528ea59ac2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41737 } }
I20260927 18:46:04.605829 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:04.605890 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:04.605983 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:04.606742 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd7cecd826a435ba51b73528ea59ac2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41737 } }
I20260927 18:46:04.607061 5580 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd7cecd826a435ba51b73528ea59ac2; no voters:
I20260927 18:46:04.607340 5580 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:04.607429 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:04.607717 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd7cecd826a435ba51b73528ea59ac2, State: Running, Role: LEADER
I20260927 18:46:04.608000 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd7cecd826a435ba51b73528ea59ac2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41737 } }
I20260927 18:46:04.608533 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:04.608948 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd7cecd826a435ba51b73528ea59ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd7cecd826a435ba51b73528ea59ac2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41737 } } }
I20260927 18:46:04.609069 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:04.609488 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd7cecd826a435ba51b73528ea59ac2. Latest consensus state: current_term: 1 leader_uuid: "6bd7cecd826a435ba51b73528ea59ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd7cecd826a435ba51b73528ea59ac2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 41737 } } }
I20260927 18:46:04.609719 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:04.609488 5590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:04.612289 5590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:04.617096 5590 catalog_manager.cc:1347] Generated new cluster ID: 378813edb7f84daa86cdac5840d8a024
I20260927 18:46:04.617189 5590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:04.626655 5590 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:04.627404 5590 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:04.634407 5590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2: Generated new TSK 0
I20260927 18:46:04.634919 5590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:04.637363 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41737
--builtin_ntp_servers=127.18.235.148:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:46:04.758203 5604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:04.758409 5604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:04.758441 5604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:04.762265 5604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:04.762307 5604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:04.762363 5604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:46:04.766831 5604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:04.767647 5604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:04.768815 5604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:04.777210 5610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:04.777240 5609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:04.777247 5612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:04.777264 5604 server_base.cc:1034] running on GCE node
I20260927 18:46:04.777750 5604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:04.778264 5604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:04.779412 5604 hybrid_clock.cc:648] HybridClock initialized: now 1790534764779400 us; error 24 us; skew 500 ppm
I20260927 18:46:04.780907 5604 webserver.cc:460] Webserver started at http://127.18.235.129:37175/ using document root <none> and password file <none>
I20260927 18:46:04.781411 5604 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:04.781469 5604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:04.781616 5604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:04.783376 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "8ad90e55a2e64294a629a14025cb7c4e"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "ca5bd2f696bbb3cb111169dd09b10ef8"
server_key_iv: "c4a00fe5a440f9b7249fb1af7e35de04"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.783839 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "8ad90e55a2e64294a629a14025cb7c4e"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "ca5bd2f696bbb3cb111169dd09b10ef8"
server_key_iv: "c4a00fe5a440f9b7249fb1af7e35de04"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.787011 5604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:04.789034 5618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.790144 5604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:04.790236 5604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "8ad90e55a2e64294a629a14025cb7c4e"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "ca5bd2f696bbb3cb111169dd09b10ef8"
server_key_iv: "c4a00fe5a440f9b7249fb1af7e35de04"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.790326 5604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:04.804266 5604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:04.804800 5604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:04.804957 5604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:04.805524 5604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:04.806356 5604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:04.806404 5604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.806449 5604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:04.806481 5604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.815137 5604 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:46109
I20260927 18:46:04.815192 5731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:46109 every 8 connection(s)
I20260927 18:46:04.815907 5604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:04.823061 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5604
I20260927 18:46:04.823191 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:04.823475 19374 external_mini_cluster.cc:1428] Setting key e071f8dcbc9199e13b3b43f7239b24d2
I20260927 18:46:04.824162 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.817470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:59543 (local address 127.18.235.190:41737)
0927 18:46:04.817604 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.817610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.818574 (+ 964us) server_negotiation.cc:407] Connection header received
0927 18:46:04.819819 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.819821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.819873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.819950 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.820715 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.821285 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.822358 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.822585 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.822668 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.823716 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.823719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.823721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:04.823733 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.823753 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:04.823762 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.823833 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.823963 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.824029 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:46:04.824975 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.817861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41737 (local address 127.18.235.129:59543)
0927 18:46:04.818440 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.818476 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.819638 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.820089 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.820096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.820175 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:04.820587 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.820621 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.821400 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.821403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.822250 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:04.822259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:04.822705 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:04.822708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:04.822848 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:04.823438 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:04.823440 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:04.823487 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:04.823561 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:04.823843 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:04.823850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:04.823855 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.823944 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.824181 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20260927 18:46:04.825495 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41737
--builtin_ntp_servers=127.18.235.148:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:46:04.826095 5732 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41737
I20260927 18:46:04.826295 5732 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:04.826788 5732 heartbeater.cc:511] Master 127.18.235.190:41737 requested a full tablet report, sending...
I20260927 18:46:04.828270 5544 ts_manager.cc:194] Registered new tserver with Master: 8ad90e55a2e64294a629a14025cb7c4e (127.18.235.129:46109)
I20260927 18:46:04.829730 5544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.129:59543
W20260927 18:46:04.928929 5735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:04.929154 5735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:04.929189 5735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:04.932934 5735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:04.932976 5735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:04.933023 5735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:46:04.937405 5735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:04.938211 5735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:04.939328 5735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:04.947829 5735 server_base.cc:1034] running on GCE node
W20260927 18:46:04.947865 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:04.947865 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:04.948056 5743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:04.948400 5735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:04.948976 5735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:04.950137 5735 hybrid_clock.cc:648] HybridClock initialized: now 1790534764950120 us; error 34 us; skew 500 ppm
I20260927 18:46:04.951732 5735 webserver.cc:460] Webserver started at http://127.18.235.130:45599/ using document root <none> and password file <none>
I20260927 18:46:04.952232 5735 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:04.952298 5735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:04.952471 5735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:04.954334 5735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "d4ac6a9709214a43b37362a2d78a647a"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "f448a0e6185c42dd337b68bd39951251"
server_key_iv: "11518602887da3caef3e859d57e1ddfb"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.954850 5735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "d4ac6a9709214a43b37362a2d78a647a"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "f448a0e6185c42dd337b68bd39951251"
server_key_iv: "11518602887da3caef3e859d57e1ddfb"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.958187 5735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:04.960325 5749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.961416 5735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:04.961519 5735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "d4ac6a9709214a43b37362a2d78a647a"
format_stamp: "Formatted at 2026-09-27 18:46:04 on dist-test-slave-6lr6"
server_key: "f448a0e6185c42dd337b68bd39951251"
server_key_iv: "11518602887da3caef3e859d57e1ddfb"
server_key_version: "encryptionkey@0"
I20260927 18:46:04.961608 5735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:05.002560 5735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:05.003153 5735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:05.003317 5735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:05.003850 5735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:05.004724 5735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:05.004776 5735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:05.004830 5735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:05.004863 5735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:05.013661 5735 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:43007
I20260927 18:46:05.013727 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:43007 every 8 connection(s)
I20260927 18:46:05.014494 5735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:05.021639 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5735
I20260927 18:46:05.021740 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:05.022003 19374 external_mini_cluster.cc:1428] Setting key de628acc327668f71951429713bf387b
I20260927 18:46:05.022806 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.016108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:33761 (local address 127.18.235.190:41737)
0927 18:46:05.016225 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.016229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.017231 (+ 1002us) server_negotiation.cc:407] Connection header received
0927 18:46:05.018492 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.018495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.018545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.018627 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:05.019424 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.019952 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.020999 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.021287 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.021378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.022330 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:05.022335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:05.022338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:05.022354 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:05.022379 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:05.022392 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:05.022457 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.022619 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.022679 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:05.023643 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.016465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41737 (local address 127.18.235.130:33761)
0927 18:46:05.017057 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.017130 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.018311 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.018764 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.018771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.018849 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:05.019304 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.019316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.020050 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.020052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.020896 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.020903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.021418 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.021422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.021555 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.022093 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:05.022095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:05.022132 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:05.022188 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:05.022469 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:05.022476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:05.022482 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.022589 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.022826 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260927 18:46:05.023976 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41737
--builtin_ntp_servers=127.18.235.148:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:46:05.024741 5863 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41737
I20260927 18:46:05.024931 5863 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:05.025429 5863 heartbeater.cc:511] Master 127.18.235.190:41737 requested a full tablet report, sending...
I20260927 18:46:05.026405 5544 ts_manager.cc:194] Registered new tserver with Master: d4ac6a9709214a43b37362a2d78a647a (127.18.235.130:43007)
I20260927 18:46:05.026948 5544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.130:33761
W20260927 18:46:05.128746 5866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:05.128965 5866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:05.129002 5866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:05.132812 5866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:05.132859 5866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:05.132911 5866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:46:05.137360 5866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:41737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:05.138171 5866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:05.139323 5866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:05.147769 5874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:05.147773 5872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:05.147758 5871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:05.148345 5866 server_base.cc:1034] running on GCE node
I20260927 18:46:05.148679 5866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:05.149277 5866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:05.150458 5866 hybrid_clock.cc:648] HybridClock initialized: now 1790534765150431 us; error 42 us; skew 500 ppm
I20260927 18:46:05.152062 5866 webserver.cc:460] Webserver started at http://127.18.235.131:36669/ using document root <none> and password file <none>
I20260927 18:46:05.152546 5866 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:05.152611 5866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:05.152799 5866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:05.154680 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "045a1906023d4ef78634934520dd91bb"
format_stamp: "Formatted at 2026-09-27 18:46:05 on dist-test-slave-6lr6"
server_key: "1cc69815694a30b4cf30803f3f99933c"
server_key_iv: "73f50778188db2ebaee6e3363d7349bf"
server_key_version: "encryptionkey@0"
I20260927 18:46:05.155208 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "045a1906023d4ef78634934520dd91bb"
format_stamp: "Formatted at 2026-09-27 18:46:05 on dist-test-slave-6lr6"
server_key: "1cc69815694a30b4cf30803f3f99933c"
server_key_iv: "73f50778188db2ebaee6e3363d7349bf"
server_key_version: "encryptionkey@0"
I20260927 18:46:05.158617 5866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:05.160812 5880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:05.161957 5866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:05.162055 5866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "045a1906023d4ef78634934520dd91bb"
format_stamp: "Formatted at 2026-09-27 18:46:05 on dist-test-slave-6lr6"
server_key: "1cc69815694a30b4cf30803f3f99933c"
server_key_iv: "73f50778188db2ebaee6e3363d7349bf"
server_key_version: "encryptionkey@0"
I20260927 18:46:05.162138 5866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:05.177234 5866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:05.177779 5866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:05.177927 5866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:05.178414 5866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:05.179232 5866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:05.179277 5866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:05.179325 5866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:05.179353 5866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:05.187287 5866 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:33365
I20260927 18:46:05.187319 5993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:33365 every 8 connection(s)
I20260927 18:46:05.188045 5866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:05.190231 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 5866
I20260927 18:46:05.190322 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:05.190558 19374 external_mini_cluster.cc:1428] Setting key 36ecb23f43601a9ee51aaa1515b3b916
I20260927 18:46:05.196607 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.189616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:55753 (local address 127.18.235.190:41737)
0927 18:46:05.189777 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.189782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.190762 (+ 980us) server_negotiation.cc:407] Connection header received
0927 18:46:05.192372 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.192376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.192429 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.192491 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:05.193417 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.193952 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.195056 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.195275 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.195351 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.196234 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:05.196237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:05.196239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:05.196251 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:05.196270 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:05.196277 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:05.196338 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.196438 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.196485 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:46:05.197381 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.189977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:41737 (local address 127.18.235.131:55753)
0927 18:46:05.190616 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.190662 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.192159 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.192670 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.192685 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.192769 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:05.193295 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.193309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.194048 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.194050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.194955 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.194964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.195385 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.195387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.195520 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.196010 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:05.196011 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:05.196049 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:05.196106 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:05.196344 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:05.196349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:05.196352 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.196434 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.196628 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260927 18:46:05.198297 5994 heartbeater.cc:347] Connected to a master server at 127.18.235.190:41737
I20260927 18:46:05.198489 5994 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:05.198944 5994 heartbeater.cc:511] Master 127.18.235.190:41737 requested a full tablet report, sending...
I20260927 18:46:05.199812 5544 ts_manager.cc:194] Registered new tserver with Master: 045a1906023d4ef78634934520dd91bb (127.18.235.131:33365)
I20260927 18:46:05.200331 5544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.131:55753
I20260927 18:46:05.203860 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:05.208631 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.205561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.18.235.190:41737)
0927 18:46:05.205685 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.205689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.205807 (+ 118us) server_negotiation.cc:407] Connection header received
0927 18:46:05.205885 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.205887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.205920 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.205977 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:05.206189 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.206639 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.207361 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.207547 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.207607 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.208244 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:05.208246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:05.208248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:05.208258 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:05.208274 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:05.208281 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:05.208367 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.208506 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.208541 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:46:05.210945 5544 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:05.213673 5544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:05.226182 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.221626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.18.235.131:33365)
0927 18:46:05.222677 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.222688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.222707 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:46:05.222764 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.222768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.222919 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.223045 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:05.223250 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.224030 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.225082 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.225600 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.225701 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.225853 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.225972 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.226052 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":121,"threads_started":1}
I20260927 18:46:05.226183 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.220856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:33365 (local address 127.0.0.1:38824)
0927 18:46:05.221249 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.221261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.221340 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.223046 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.223048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.223066 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:05.223142 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.223147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.224130 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.224133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.224925 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.224934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.225700 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.225704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.225803 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.225835 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.225928 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.226036 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260927 18:46:05.226330 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.220719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:43007 (local address 127.0.0.1:35510)
0927 18:46:05.221184 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.221230 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.221340 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.222069 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.222076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.222112 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:05.222236 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.222242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.223623 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.223626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.224953 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.224963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.225985 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.225989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.226075 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.226098 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.226140 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.226203 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":56,"thread_start_us":75,"threads_started":1}
I20260927 18:46:05.226994 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.220867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.18.235.130:43007)
0927 18:46:05.221600 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.221610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.221633 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:46:05.221704 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.221709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.221903 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.222380 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:05.222511 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.223524 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.225078 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.225863 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.226592 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.226731 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.226775 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.226847 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":109,"threads_started":1}
I20260927 18:46:05.227059 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.221384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:46109 (local address 127.0.0.1:54774)
0927 18:46:05.221919 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.221932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.222006 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.222436 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.222438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.222453 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:05.222515 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.222519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.223548 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.223551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.225596 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.225606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.226696 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.226700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.226782 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.226803 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.226874 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.226937 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":101,"thread_start_us":77,"threads_started":1}
I20260927 18:46:05.227145 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.221545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.18.235.129:46109)
0927 18:46:05.222029 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.222040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.222061 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:46:05.222127 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.222131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.222307 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.222428 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:05.222617 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.223461 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.225728 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.226452 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.226552 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.226705 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.226915 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.227000 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":108,"threads_started":1}
I20260927 18:46:05.229274 5928 tablet_service.cc:1467] Processing CreateTablet for tablet c3ec92d7039a47ba889c2acc53eb4a39 (DEFAULT_TABLE table=test-table [id=7d02a1b1c34d4f7d91f8269f2aa87404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:05.229879 5797 tablet_service.cc:1467] Processing CreateTablet for tablet c3ec92d7039a47ba889c2acc53eb4a39 (DEFAULT_TABLE table=test-table [id=7d02a1b1c34d4f7d91f8269f2aa87404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:05.230338 5928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ec92d7039a47ba889c2acc53eb4a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:05.230799 5797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ec92d7039a47ba889c2acc53eb4a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:05.230638 5666 tablet_service.cc:1467] Processing CreateTablet for tablet c3ec92d7039a47ba889c2acc53eb4a39 (DEFAULT_TABLE table=test-table [id=7d02a1b1c34d4f7d91f8269f2aa87404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:05.231336 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ec92d7039a47ba889c2acc53eb4a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:05.237329 6011 tablet_bootstrap.cc:492] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: Bootstrap starting.
I20260927 18:46:05.237762 6013 tablet_bootstrap.cc:492] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Bootstrap starting.
I20260927 18:46:05.237996 6012 tablet_bootstrap.cc:492] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Bootstrap starting.
I20260927 18:46:05.239506 6012 tablet_bootstrap.cc:654] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:05.239529 6013 tablet_bootstrap.cc:654] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:05.239530 6011 tablet_bootstrap.cc:654] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:05.240144 6012 log.cc:824] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:05.240300 6013 log.cc:824] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:05.240300 6011 log.cc:824] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:05.241868 6012 tablet_bootstrap.cc:492] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: No bootstrap required, opened a new log
I20260927 18:46:05.242043 6012 ts_tablet_manager.cc:1397] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:05.242095 6013 tablet_bootstrap.cc:492] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: No bootstrap required, opened a new log
I20260927 18:46:05.242086 6011 tablet_bootstrap.cc:492] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: No bootstrap required, opened a new log
I20260927 18:46:05.242246 6011 ts_tablet_manager.cc:1397] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:46:05.242246 6013 ts_tablet_manager.cc:1397] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:46:05.245005 6012 raft_consensus.cc:348] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.245239 6012 raft_consensus.cc:374] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:05.245317 6012 raft_consensus.cc:729] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ac6a9709214a43b37362a2d78a647a, State: Initialized, Role: FOLLOWER
I20260927 18:46:05.245719 6012 consensus_queue.cc:260] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.246275 5863 heartbeater.cc:503] Master 127.18.235.190:41737 was elected leader, sending a full tablet report...
I20260927 18:46:05.246151 6011 raft_consensus.cc:348] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.246151 6013 raft_consensus.cc:348] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.246367 6013 raft_consensus.cc:374] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:05.246367 6011 raft_consensus.cc:374] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:05.246412 6011 raft_consensus.cc:729] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad90e55a2e64294a629a14025cb7c4e, State: Initialized, Role: FOLLOWER
I20260927 18:46:05.246419 6013 raft_consensus.cc:729] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045a1906023d4ef78634934520dd91bb, State: Initialized, Role: FOLLOWER
I20260927 18:46:05.246605 6012 ts_tablet_manager.cc:1428] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:05.246841 6013 consensus_queue.cc:260] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.246841 6011 consensus_queue.cc:260] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.247416 5994 heartbeater.cc:503] Master 127.18.235.190:41737 was elected leader, sending a full tablet report...
I20260927 18:46:05.247629 5732 heartbeater.cc:503] Master 127.18.235.190:41737 was elected leader, sending a full tablet report...
I20260927 18:46:05.247709 6013 ts_tablet_manager.cc:1428] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:05.248574 6011 ts_tablet_manager.cc:1428] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260927 18:46:05.265296 5864 tablet.cc:2369] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:05.265820 6017 raft_consensus.cc:482] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:05.265944 6017 raft_consensus.cc:504] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.266804 6017 leader_election.cc:288] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 045a1906023d4ef78634934520dd91bb (127.18.235.131:33365), 8ad90e55a2e64294a629a14025cb7c4e (127.18.235.129:46109)
I20260927 18:46:05.270663 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.266969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:33365 (local address 127.18.235.130:47827)
0927 18:46:05.267079 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.267106 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.267183 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.267493 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.267495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.267511 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:05.267609 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.267615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.268471 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.268473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.269619 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.269630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.270350 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.270353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.270424 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.270443 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.270496 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.270547 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:46:05.270699 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.267026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:47827 (local address 127.18.235.131:33365)
0927 18:46:05.267126 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.267130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.267143 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:05.267289 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.267292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.267335 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.267396 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:05.267716 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.268383 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.269750 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.270242 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.270337 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.270380 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.270529 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.270593 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:46:05.271034 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.267217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:46109 (local address 127.18.235.130:41111)
0927 18:46:05.267500 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.267511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:05.267603 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:05.267830 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:05.267833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:05.267848 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:05.267955 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.267961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.268833 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.268837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.269912 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:05.269920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.270630 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.270633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:05.270718 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.270740 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:05.270849 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 18:46:05.270909 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260927 18:46:05.271035 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.267292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:41111 (local address 127.18.235.129:46109)
0927 18:46:05.267403 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.267407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.267583 (+ 176us) server_negotiation.cc:407] Connection header received
0927 18:46:05.267703 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.267706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.267748 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.267808 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:05.268187 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.268738 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.270012 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.270516 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.270640 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.270698 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.270884 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.270940 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:05.271297 5948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ec92d7039a47ba889c2acc53eb4a39" candidate_uuid: "d4ac6a9709214a43b37362a2d78a647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045a1906023d4ef78634934520dd91bb" is_pre_election: true
I20260927 18:46:05.271528 5948 raft_consensus.cc:2454] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ac6a9709214a43b37362a2d78a647a in term 0.
I20260927 18:46:05.271642 5686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ec92d7039a47ba889c2acc53eb4a39" candidate_uuid: "d4ac6a9709214a43b37362a2d78a647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad90e55a2e64294a629a14025cb7c4e" is_pre_election: true
I20260927 18:46:05.271863 5686 raft_consensus.cc:2454] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ac6a9709214a43b37362a2d78a647a in term 0.
I20260927 18:46:05.271962 5753 leader_election.cc:302] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045a1906023d4ef78634934520dd91bb, d4ac6a9709214a43b37362a2d78a647a; no voters:
I20260927 18:46:05.272192 6017 raft_consensus.cc:2781] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:05.272265 6017 raft_consensus.cc:482] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:05.272315 6017 raft_consensus.cc:3037] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:05.273164 6017 raft_consensus.cc:504] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.273437 6017 leader_election.cc:288] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [CANDIDATE]: Term 1 election: Requested vote from peers 045a1906023d4ef78634934520dd91bb (127.18.235.131:33365), 8ad90e55a2e64294a629a14025cb7c4e (127.18.235.129:46109)
I20260927 18:46:05.273720 5948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ec92d7039a47ba889c2acc53eb4a39" candidate_uuid: "d4ac6a9709214a43b37362a2d78a647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045a1906023d4ef78634934520dd91bb"
I20260927 18:46:05.273773 5686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ec92d7039a47ba889c2acc53eb4a39" candidate_uuid: "d4ac6a9709214a43b37362a2d78a647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad90e55a2e64294a629a14025cb7c4e"
I20260927 18:46:05.273829 5948 raft_consensus.cc:3037] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:05.273865 5686 raft_consensus.cc:3037] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:05.274930 5686 raft_consensus.cc:2454] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ac6a9709214a43b37362a2d78a647a in term 1.
I20260927 18:46:05.274930 5948 raft_consensus.cc:2454] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ac6a9709214a43b37362a2d78a647a in term 1.
I20260927 18:46:05.275180 5753 leader_election.cc:302] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045a1906023d4ef78634934520dd91bb, d4ac6a9709214a43b37362a2d78a647a; no voters:
I20260927 18:46:05.275310 6017 raft_consensus.cc:2781] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:05.275494 6017 raft_consensus.cc:686] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 1 LEADER]: Becoming Leader. State: Replica: d4ac6a9709214a43b37362a2d78a647a, State: Running, Role: LEADER
I20260927 18:46:05.275753 6017 consensus_queue.cc:237] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } }
I20260927 18:46:05.278319 5544 catalog_manager.cc:5495] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a reported cstate change: term changed from 0 to 1, leader changed from <none> to d4ac6a9709214a43b37362a2d78a647a (127.18.235.130). New cstate: current_term: 1 leader_uuid: "d4ac6a9709214a43b37362a2d78a647a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ac6a9709214a43b37362a2d78a647a" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:05.287206 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.284364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.18.235.130:43007)
0927 18:46:05.284453 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:05.284456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:05.284466 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:46:05.284540 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:05.284542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:05.284585 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:05.284658 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:05.284825 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.285340 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.286127 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:05.286328 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:05.286389 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:05.286597 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:05.286921 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:05.287016 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:05.287072 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:46:05.287098 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:46:05.291941 5686 raft_consensus.cc:1261] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e [term 1 FOLLOWER]: Refusing update from remote peer d4ac6a9709214a43b37362a2d78a647a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:05.291965 5948 raft_consensus.cc:1261] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 1 FOLLOWER]: Refusing update from remote peer d4ac6a9709214a43b37362a2d78a647a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:05.292512 6017 consensus_queue.cc:1035] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [LEADER]: Connected to new peer: Peer: permanent_uuid: "045a1906023d4ef78634934520dd91bb" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:05.292670 6021 consensus_queue.cc:1035] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad90e55a2e64294a629a14025cb7c4e" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:05.298897 6025 mvcc.cc:205] Tried to move back new op lower bound from 7334030398630248448 to 7334030398571413504. Current Snapshot: MvccSnapshot[applied={T|T < 7334030398630248448}]
I20260927 18:46:05.305891 5542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:05.306008 5542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:05.308053 5542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bd7cecd826a435ba51b73528ea59ac2: Sending DeleteTablet for 3 replicas of tablet c3ec92d7039a47ba889c2acc53eb4a39
I20260927 18:46:05.308707 5797 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ec92d7039a47ba889c2acc53eb4a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:05 UTC) from {username='slave'} at 127.0.0.1:35510
I20260927 18:46:05.308833 5928 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ec92d7039a47ba889c2acc53eb4a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:05 UTC) from {username='slave'} at 127.0.0.1:38824
I20260927 18:46:05.309252 5666 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ec92d7039a47ba889c2acc53eb4a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:05 UTC) from {username='slave'} at 127.0.0.1:54774
I20260927 18:46:05.309499 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5604
I20260927 18:46:05.309672 6035 tablet_replica.cc:330] T c3ec92d7039a47ba889c2acc53eb4a39 P 8ad90e55a2e64294a629a14025cb7c4e: stopping tablet replica
I20260927 18:46:05.310140 6034 tablet_replica.cc:330] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: stopping tablet replica
I20260927 18:46:05.310388 6034 raft_consensus.cc:2229] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:05.310634 6034 raft_consensus.cc:2258] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:05.310953 6033 tablet_replica.cc:330] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: stopping tablet replica
I20260927 18:46:05.311183 6033 raft_consensus.cc:2229] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:05.311455 6033 raft_consensus.cc:2258] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:05.312920 6033 ts_tablet_manager.cc:1905] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:05.312944 6034 ts_tablet_manager.cc:1905] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:05.315348 6033 ts_tablet_manager.cc:1918] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:05.315423 6033 log.cc:1196] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/c3ec92d7039a47ba889c2acc53eb4a39
I20260927 18:46:05.315424 6034 ts_tablet_manager.cc:1918] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:05.315498 6034 log.cc:1196] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/c3ec92d7039a47ba889c2acc53eb4a39
I20260927 18:46:05.315691 6033 ts_tablet_manager.cc:1939] T c3ec92d7039a47ba889c2acc53eb4a39 P d4ac6a9709214a43b37362a2d78a647a: Deleting consensus metadata
I20260927 18:46:05.315776 6034 ts_tablet_manager.cc:1939] T c3ec92d7039a47ba889c2acc53eb4a39 P 045a1906023d4ef78634934520dd91bb: Deleting consensus metadata
I20260927 18:46:05.316761 5531 catalog_manager.cc:4841] TS 045a1906023d4ef78634934520dd91bb (127.18.235.131:33365): tablet c3ec92d7039a47ba889c2acc53eb4a39 (table test-table [id=7d02a1b1c34d4f7d91f8269f2aa87404]) successfully deleted
I20260927 18:46:05.317082 5530 catalog_manager.cc:4841] TS d4ac6a9709214a43b37362a2d78a647a (127.18.235.130:43007): tablet c3ec92d7039a47ba889c2acc53eb4a39 (table test-table [id=7d02a1b1c34d4f7d91f8269f2aa87404]) successfully deleted
W20260927 18:46:05.318257 5531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:46109 (error 108)
I20260927 18:46:05.318921 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5735
I20260927 18:46:05.319095 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:05.318859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:46109 (local address 127.0.0.1:54778)
0927 18:46:05.318965 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:05.319027 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:46:05.320350 5531 catalog_manager.cc:4584] TS 8ad90e55a2e64294a629a14025cb7c4e (127.18.235.129:46109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3ec92d7039a47ba889c2acc53eb4a39: Network error: Client connection negotiation failed: client connection to 127.18.235.129:46109: connect: Connection refused (error 111)
I20260927 18:46:05.323803 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5866
I20260927 18:46:05.328037 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 5513
2026-09-27T18:46:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (921 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:05 dist-test-slave-6lr6 krb5kdc[6039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:05 dist-test-slave-6lr6 krb5kdc[6039](info): set up 2 sockets
Sep 27 18:46:05 dist-test-slave-6lr6 krb5kdc[6039](info): commencing operation
krb5kdc: starting...
W20260927 18:46:06.276770 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:06.381946 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:43353
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:43353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:06.486384 6052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:06.486604 6052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:06.486657 6052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:06.490513 6052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:06.490559 6052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:06.490584 6052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:06.490607 6052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:06.490635 6052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:06.495484 6052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:43353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:06.496340 6052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:06.497210 6052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:06.504901 6058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.504895 6060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.504892 6057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:06.505129 6052 server_base.cc:1034] running on GCE node
I20260927 18:46:06.505635 6052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:06.506567 6052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:06.507722 6052 hybrid_clock.cc:648] HybridClock initialized: now 1790534766507708 us; error 32 us; skew 500 ppm
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:06.510165 6052 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:06.510999 6052 webserver.cc:460] Webserver started at http://127.18.235.190:40589/ using document root <none> and password file <none>
I20260927 18:46:06.511466 6052 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:06.511525 6052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:06.511710 6052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:06.513513 6052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "0c3db74e600446f28757a748d744df06"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "3b67ac2d669e8a2df36c02dc342c6ca7"
server_key_iv: "34636695f1176ccf7b2fd1ed655f255a"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.514005 6052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "0c3db74e600446f28757a748d744df06"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "3b67ac2d669e8a2df36c02dc342c6ca7"
server_key_iv: "34636695f1176ccf7b2fd1ed655f255a"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.517233 6052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:06.519263 6067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.520552 6052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:06.520668 6052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "0c3db74e600446f28757a748d744df06"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "3b67ac2d669e8a2df36c02dc342c6ca7"
server_key_iv: "34636695f1176ccf7b2fd1ed655f255a"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.520761 6052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:06.530839 6052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:06.533268 6052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:06.533413 6052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:06.539536 6052 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:43353
I20260927 18:46:06.539580 6119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:43353 every 8 connection(s)
I20260927 18:46:06.540323 6052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:06.542805 6120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:06.546555 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6052
I20260927 18:46:06.546684 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:06.546936 19374 external_mini_cluster.cc:1428] Setting key 114d86074cb4a007d94628f61e06468d
I20260927 18:46:06.548092 6120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: Bootstrap starting.
I20260927 18:46:06.550235 6120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:06.550940 6120 log.cc:824] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:06.553313 6120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: No bootstrap required, opened a new log
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:06.556237 6120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3db74e600446f28757a748d744df06" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43353 } }
I20260927 18:46:06.556488 6120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:06.556540 6120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3db74e600446f28757a748d744df06, State: Initialized, Role: FOLLOWER
I20260927 18:46:06.556911 6120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3db74e600446f28757a748d744df06" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43353 } }
I20260927 18:46:06.557029 6120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:06.557085 6120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:06.557190 6120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:06.557888 6120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3db74e600446f28757a748d744df06" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43353 } }
I20260927 18:46:06.558171 6120 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3db74e600446f28757a748d744df06; no voters:
I20260927 18:46:06.558406 6120 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:06.558497 6125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:06.558748 6125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [term 1 LEADER]: Becoming Leader. State: Replica: 0c3db74e600446f28757a748d744df06, State: Running, Role: LEADER
I20260927 18:46:06.559073 6125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3db74e600446f28757a748d744df06" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43353 } }
I20260927 18:46:06.559510 6120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:06.560127 6126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3db74e600446f28757a748d744df06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3db74e600446f28757a748d744df06" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43353 } } }
I20260927 18:46:06.560175 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.548087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.18.235.190:43353)
0927 18:46:06.548445 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:06.548460 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:06.548492 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:46:06.549057 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:06.549071 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:06.549415 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:06.549670 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:06.550470 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.551415 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.552207 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.552621 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.552754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.555124 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:06.555128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:06.555139 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:06.555163 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:06.557221 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.557622 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.557628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.557633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.557737 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.557980 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.557991 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.557993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.558380 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:06.558570 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:06.558845 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 18:46:06.559259 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260927 18:46:06.560146 6127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3db74e600446f28757a748d744df06. Latest consensus state: current_term: 1 leader_uuid: "0c3db74e600446f28757a748d744df06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3db74e600446f28757a748d744df06" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 43353 } } }
I20260927 18:46:06.560257 6126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:06.560273 6127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06 [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:06.563844 6140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:06.563920 6140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:06.563971 6132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:06.564779 6132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:06.569610 6132 catalog_manager.cc:1347] Generated new cluster ID: 0f99706a1b78460699ad4c7d3e66a214
I20260927 18:46:06.569658 6132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:06.576787 6132 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:06.577643 6132 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:06.584372 6132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: Generated new TSK 0
I20260927 18:46:06.584883 6132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
Entry for principal kudu/127.18.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab.
I20260927 18:46:06.611822 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43353
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:06.737684 6146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:06.737886 6146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:06.737923 6146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:06.741705 6146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:06.741750 6146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:06.741803 6146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:46:06.746676 6146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.129.keytab
--principal=kudu/127.18.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:06.747498 6146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:06.748339 6146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:06.756742 6152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.756671 6154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.756770 6151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:06.756963 6146 server_base.cc:1034] running on GCE node
I20260927 18:46:06.757378 6146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:06.757964 6146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:06.759141 6146 hybrid_clock.cc:648] HybridClock initialized: now 1790534766759118 us; error 33 us; skew 500 ppm
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:06.761888 6146 init.cc:377] Logged in from keytab as kudu/127.18.235.129@KRBTEST.COM (short username kudu)
I20260927 18:46:06.762684 6146 webserver.cc:460] Webserver started at http://127.18.235.129:39861/ using document root <none> and password file <none>
I20260927 18:46:06.763190 6146 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:06.763255 6146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:06.763448 6146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:06.765415 6146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "005c837d15484b489c90705841b7376b"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "740d64cf82e509c2dd1167eb497eb6df"
server_key_iv: "62438b47f4f5e0ccbe55c305a1bbe3a2"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.765950 6146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "005c837d15484b489c90705841b7376b"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "740d64cf82e509c2dd1167eb497eb6df"
server_key_iv: "62438b47f4f5e0ccbe55c305a1bbe3a2"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.769465 6146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:06.771631 6161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.772677 6146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:06.772781 6146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "005c837d15484b489c90705841b7376b"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "740d64cf82e509c2dd1167eb497eb6df"
server_key_iv: "62438b47f4f5e0ccbe55c305a1bbe3a2"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.772871 6146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:06.787120 6146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:06.789675 6146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:06.789839 6146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:06.790328 6146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:06.791049 6146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:06.791095 6146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.791141 6146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:06.791164 6146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.798815 6146 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:44433
I20260927 18:46:06.798839 6274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:44433 every 8 connection(s)
I20260927 18:46:06.799547 6146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:06.807343 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6146
I20260927 18:46:06.807466 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:06.807796 19374 external_mini_cluster.cc:1428] Setting key 5e274ee5a8cf23e8f73b4dc163549cf5
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.129@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:06.812176 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.801065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:47123 (local address 127.18.235.190:43353)
0927 18:46:06.801180 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:06.801185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:06.801903 (+ 718us) server_negotiation.cc:407] Connection header received
0927 18:46:06.803122 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:06.803126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:06.803182 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:06.803252 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:06.804298 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.804856 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.805978 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.806214 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.806302 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.808667 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:06.808671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:06.808674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:06.808700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:06.810149 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.810678 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.810681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.810683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.810788 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.811128 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.811132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.811134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.811359 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:06.811459 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:06.811884 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 18:46:06.812008 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:06.812785 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.801381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43353 (local address 127.18.235.129:47123)
0927 18:46:06.801770 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:06.801807 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:06.802922 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:06.803414 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:06.803420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:06.803717 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:06.804141 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:06.804153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.804972 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.804975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:06.805857 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:06.805867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.806343 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.806346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:06.806494 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.806978 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:06.806994 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:06.808503 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:06.810262 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:06.810268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:06.810281 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:06.810581 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:06.810878 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:06.810880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:06.810881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:06.811046 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:06.811457 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:06.811461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:06.811655 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 18:46:06.811804 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:46:06.812014 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":103,"threads_started":1}
I20260927 18:46:06.814081 6275 heartbeater.cc:347] Connected to a master server at 127.18.235.190:43353
I20260927 18:46:06.814278 6275 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:06.814672 6275 heartbeater.cc:511] Master 127.18.235.190:43353 requested a full tablet report, sending...
I20260927 18:46:06.816174 6084 ts_manager.cc:194] Registered new tserver with Master: 005c837d15484b489c90705841b7376b (127.18.235.129:44433)
WARNING: no policy specified for kudu/127.18.235.130@KRBTEST.COM; defaulting to no policy
I20260927 18:46:06.817739 6084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.129@KRBTEST.COM'} at 127.18.235.129:47123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
Entry for principal kudu/127.18.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab.
I20260927 18:46:06.833165 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43353
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:06.934352 6280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:06.934562 6280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:06.934625 6280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:06.938508 6280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:06.938555 6280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:06.938640 6280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:46:06.943476 6280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.130.keytab
--principal=kudu/127.18.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:06.944291 6280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:06.945159 6280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:06.953348 6288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.953488 6285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:06.953574 6280 server_base.cc:1034] running on GCE node
W20260927 18:46:06.953359 6286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:06.953954 6280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:06.954487 6280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:06.955660 6280 hybrid_clock.cc:648] HybridClock initialized: now 1790534766955622 us; error 27 us; skew 500 ppm
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:06.958120 6280 init.cc:377] Logged in from keytab as kudu/127.18.235.130@KRBTEST.COM (short username kudu)
I20260927 18:46:06.958830 6280 webserver.cc:460] Webserver started at http://127.18.235.130:33773/ using document root <none> and password file <none>
I20260927 18:46:06.959344 6280 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:06.959405 6280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:06.959582 6280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:06.961488 6280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "b37472d8233d4a959b39f5461538dffe"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "dc0ef4621722984151121f57852a1e66"
server_key_iv: "1a31a9223be9c9f87b86147b4e1485e6"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.961966 6280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "b37472d8233d4a959b39f5461538dffe"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "dc0ef4621722984151121f57852a1e66"
server_key_iv: "1a31a9223be9c9f87b86147b4e1485e6"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.965161 6280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:06.967201 6295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.968240 6280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:46:06.968335 6280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "b37472d8233d4a959b39f5461538dffe"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-6lr6"
server_key: "dc0ef4621722984151121f57852a1e66"
server_key_iv: "1a31a9223be9c9f87b86147b4e1485e6"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.968416 6280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:06.979000 6280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:06.981631 6280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:06.981781 6280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:06.982259 6280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:06.982939 6280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:06.982982 6280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.983027 6280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:06.983057 6280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.990547 6280 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:39243
I20260927 18:46:06.990622 6408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:39243 every 8 connection(s)
I20260927 18:46:06.991286 6280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:06.997496 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6280
I20260927 18:46:06.997612 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:06.997851 19374 external_mini_cluster.cc:1428] Setting key f624de483d08b26b7b38357daf00344c
Sep 27 18:46:06 dist-test-slave-6lr6 krb5kdc[6039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.130@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:07.003180 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.992760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:34925 (local address 127.18.235.190:43353)
0927 18:46:06.992871 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:06.992874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:06.993574 (+ 700us) server_negotiation.cc:407] Connection header received
0927 18:46:06.994803 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:06.994806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:06.994865 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:06.994935 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:06.995938 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.996469 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.997614 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.997870 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.997962 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.000170 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:07.000172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:07.000174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:07.000197 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:07.001425 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.001861 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.001864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.001865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.001936 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.002230 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.002234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.002235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.002400 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:07.002481 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.002914 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.003029 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:07.003837 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.993017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43353 (local address 127.18.235.130:34925)
0927 18:46:06.993435 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:06.993472 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:06.994611 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:06.995098 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:06.995105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:06.995390 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:06.995803 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:06.995816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.996589 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.996592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:06.997495 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:06.997505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.998016 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.998021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:06.998166 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.998641 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:06.998660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:07.000018 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:07.001524 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:07.001530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:07.001542 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:07.001771 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:07.002015 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:07.002019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:07.002020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:07.002148 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:07.002493 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:07.002499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:07.002707 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.002856 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.003142 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260927 18:46:07.004755 6409 heartbeater.cc:347] Connected to a master server at 127.18.235.190:43353
I20260927 18:46:07.004945 6409 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:07.005380 6409 heartbeater.cc:511] Master 127.18.235.190:43353 requested a full tablet report, sending...
I20260927 18:46:07.006240 6084 ts_manager.cc:194] Registered new tserver with Master: b37472d8233d4a959b39f5461538dffe (127.18.235.130:39243)
WARNING: no policy specified for kudu/127.18.235.131@KRBTEST.COM; defaulting to no policy
I20260927 18:46:07.006750 6084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.130@KRBTEST.COM'} at 127.18.235.130:34925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
Entry for principal kudu/127.18.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab.
I20260927 18:46:07.021878 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43353
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:07.126192 6414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:07.126410 6414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:07.126451 6414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:07.130278 6414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:07.130327 6414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:07.130383 6414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:46:07.135239 6414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.131.keytab
--principal=kudu/127.18.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:07.136083 6414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:07.136955 6414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:07.145416 6422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.145424 6420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.145427 6419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.145567 6414 server_base.cc:1034] running on GCE node
I20260927 18:46:07.146032 6414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:07.146632 6414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:07.147806 6414 hybrid_clock.cc:648] HybridClock initialized: now 1790534767147781 us; error 39 us; skew 500 ppm
Sep 27 18:46:07 dist-test-slave-6lr6 krb5kdc[6039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534767, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:07.150425 6414 init.cc:377] Logged in from keytab as kudu/127.18.235.131@KRBTEST.COM (short username kudu)
I20260927 18:46:07.151144 6414 webserver.cc:460] Webserver started at http://127.18.235.131:39555/ using document root <none> and password file <none>
I20260927 18:46:07.151640 6414 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:07.151705 6414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:07.151870 6414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:07.153729 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "f9e85808a4d14a84b245240acc784d51"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "562444a2829444778c8eb15c9531dd1b"
server_key_iv: "e356248030eb71b9bc80c8bef70f84e8"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.154227 6414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "f9e85808a4d14a84b245240acc784d51"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "562444a2829444778c8eb15c9531dd1b"
server_key_iv: "e356248030eb71b9bc80c8bef70f84e8"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.157512 6414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:07.159633 6429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.160784 6414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:07.160878 6414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "f9e85808a4d14a84b245240acc784d51"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "562444a2829444778c8eb15c9531dd1b"
server_key_iv: "e356248030eb71b9bc80c8bef70f84e8"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.160961 6414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:07.178341 6414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:07.181092 6414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:07.181283 6414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:07.181811 6414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:07.182555 6414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:07.182598 6414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.182637 6414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:07.182673 6414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.190323 6414 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:33801
I20260927 18:46:07.190384 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:33801 every 8 connection(s)
I20260927 18:46:07.191120 6414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:07.196839 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6414
I20260927 18:46:07.196954 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:07.197283 19374 external_mini_cluster.cc:1428] Setting key 7c0e6e88a8be6e5da6a49b76bf1bf731
Sep 27 18:46:07 dist-test-slave-6lr6 krb5kdc[6039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534767, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.131@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:07.203681 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.192713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:44013 (local address 127.18.235.190:43353)
0927 18:46:07.192850 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.192855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.193676 (+ 821us) server_negotiation.cc:407] Connection header received
0927 18:46:07.194912 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.194915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.194960 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.195029 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:07.196096 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.196622 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.198102 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.198363 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.198461 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.200718 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:07.200723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:07.200725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:07.200752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:07.201914 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.202354 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.202357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.202359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.202431 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.202715 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.202718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.202720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.202884 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:07.202963 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.203404 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.203518 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:07.204360 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.193084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:43353 (local address 127.18.235.131:44013)
0927 18:46:07.193537 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.193577 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.194735 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.195176 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.195183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.195515 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:07.195980 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.195993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.196743 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.196747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.197970 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.197979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.198498 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.198503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.198660 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.199208 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:07.199227 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:07.200570 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:07.202011 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:07.202017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:07.202030 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:07.202268 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:07.202507 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:07.202510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:07.202512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:07.202633 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:07.202973 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:07.202979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:07.203200 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.203349 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.203629 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260927 18:46:07.205305 6543 heartbeater.cc:347] Connected to a master server at 127.18.235.190:43353
I20260927 18:46:07.205508 6543 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:07.205932 6543 heartbeater.cc:511] Master 127.18.235.190:43353 requested a full tablet report, sending...
I20260927 18:46:07.206878 6084 ts_manager.cc:194] Registered new tserver with Master: f9e85808a4d14a84b245240acc784d51 (127.18.235.131:33801)
I20260927 18:46:07.207379 6084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.235.131@KRBTEST.COM'} at 127.18.235.131:44013
I20260927 18:46:07.210469 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:07.218456 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.212148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.18.235.190:43353)
0927 18:46:07.212255 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.212259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.212430 (+ 171us) server_negotiation.cc:407] Connection header received
0927 18:46:07.212536 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.212539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.212576 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.212636 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:07.213272 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.213750 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.214535 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.214756 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.214838 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.215975 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:07.215979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:07.215981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:07.216004 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:07.217024 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.217407 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.217411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.217412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.217482 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.217715 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.217718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.217720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.217907 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:07.218021 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.218229 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.218324 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:46:07.220603 6084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:07.223395 6084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:07.236438 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.231442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.18.235.129:44433)
0927 18:46:07.231722 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.231732 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.231754 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:46:07.231826 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.231830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.232005 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.232145 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:07.232519 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.233617 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.235126 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.235909 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.236014 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.236128 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.236219 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.236303 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260927 18:46:07.236272 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.230647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:39243 (local address 127.0.0.1:47008)
0927 18:46:07.231167 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.231180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.231259 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.232350 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.232353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.232395 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:07.232460 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.232466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.233594 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.233597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.234647 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.234656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.235696 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.235700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.235844 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.235884 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.235983 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.236132 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":125,"thread_start_us":83,"threads_started":1}
I20260927 18:46:07.236513 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.230616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.18.235.131:33801)
0927 18:46:07.231491 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.231505 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.231526 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:46:07.231588 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.231593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.231796 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.232227 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:07.232343 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.233423 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.235146 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.235671 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.236141 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.236255 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.236303 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.236399 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":133,"threads_started":1}
I20260927 18:46:07.236429 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.230526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:33801 (local address 127.0.0.1:53346)
0927 18:46:07.230952 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.230997 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.231104 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.231946 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.231952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.231975 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:07.232083 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.232089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.233548 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.233552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.234999 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.235005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.235790 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.235793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.235895 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.235912 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.235970 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.236324 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":349,"thread_start_us":80,"threads_started":1}
I20260927 18:46:07.236745 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.231149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.18.235.130:39243)
0927 18:46:07.231936 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.231946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.231964 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:46:07.232028 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.232032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.232207 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.232583 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:07.232697 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.233445 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.234789 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.235568 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.235701 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.235823 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.236508 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.236613 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":104,"threads_started":1}
I20260927 18:46:07.236414 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.231269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:44433 (local address 127.0.0.1:51568)
0927 18:46:07.231607 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.231619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.231690 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.232299 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.232302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.232319 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:07.232400 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.232406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.233734 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.233738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.234992 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.235003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.236013 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.236017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.236100 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.236121 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.236187 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.236255 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":70,"thread_start_us":95,"threads_started":1}
I20260927 18:46:07.239178 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4a649f40254894951b84d9b9a388d0 (DEFAULT_TABLE table=test-table [id=820a03ea492b4de9903c986a8b75f105]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:07.239390 6477 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4a649f40254894951b84d9b9a388d0 (DEFAULT_TABLE table=test-table [id=820a03ea492b4de9903c986a8b75f105]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:07.239390 6209 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4a649f40254894951b84d9b9a388d0 (DEFAULT_TABLE table=test-table [id=820a03ea492b4de9903c986a8b75f105]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:07.240054 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4a649f40254894951b84d9b9a388d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:07.240329 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4a649f40254894951b84d9b9a388d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:07.240329 6209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4a649f40254894951b84d9b9a388d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:07.245675 6560 tablet_bootstrap.cc:492] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Bootstrap starting.
I20260927 18:46:07.246912 6561 tablet_bootstrap.cc:492] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Bootstrap starting.
I20260927 18:46:07.247294 6562 tablet_bootstrap.cc:492] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: Bootstrap starting.
I20260927 18:46:07.247867 6560 tablet_bootstrap.cc:654] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:07.249130 6560 log.cc:824] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:07.249359 6562 tablet_bootstrap.cc:654] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:07.250550 6562 log.cc:824] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:07.250823 6561 tablet_bootstrap.cc:654] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:07.251098 6560 tablet_bootstrap.cc:492] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: No bootstrap required, opened a new log
I20260927 18:46:07.251273 6560 ts_tablet_manager.cc:1397] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:46:07.251835 6561 log.cc:824] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:07.252388 6562 tablet_bootstrap.cc:492] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: No bootstrap required, opened a new log
I20260927 18:46:07.252558 6562 ts_tablet_manager.cc:1397] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:07.253299 6561 tablet_bootstrap.cc:492] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: No bootstrap required, opened a new log
I20260927 18:46:07.253460 6561 ts_tablet_manager.cc:1397] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 18:46:07.254860 6560 raft_consensus.cc:348] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.255264 6560 raft_consensus.cc:374] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:07.255357 6560 raft_consensus.cc:729] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37472d8233d4a959b39f5461538dffe, State: Initialized, Role: FOLLOWER
I20260927 18:46:07.255836 6560 consensus_queue.cc:260] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.256073 6562 raft_consensus.cc:348] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.256383 6562 raft_consensus.cc:374] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:07.256445 6562 raft_consensus.cc:729] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005c837d15484b489c90705841b7376b, State: Initialized, Role: FOLLOWER
I20260927 18:46:07.256462 6409 heartbeater.cc:503] Master 127.18.235.190:43353 was elected leader, sending a full tablet report...
I20260927 18:46:07.256340 6561 raft_consensus.cc:348] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.256567 6561 raft_consensus.cc:374] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:07.256667 6561 raft_consensus.cc:729] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e85808a4d14a84b245240acc784d51, State: Initialized, Role: FOLLOWER
I20260927 18:46:07.256830 6562 consensus_queue.cc:260] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.257047 6561 consensus_queue.cc:260] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.257776 6562 ts_tablet_manager.cc:1428] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:07.258091 6275 heartbeater.cc:503] Master 127.18.235.190:43353 was elected leader, sending a full tablet report...
I20260927 18:46:07.258498 6560 ts_tablet_manager.cc:1428] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:46:07.260198 6561 ts_tablet_manager.cc:1428] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:46:07.260627 6543 heartbeater.cc:503] Master 127.18.235.190:43353 was elected leader, sending a full tablet report...
W20260927 18:46:07.300540 6276 tablet.cc:2369] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:07.334208 6566 raft_consensus.cc:482] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:07.334395 6566 raft_consensus.cc:504] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.335453 6566 leader_election.cc:288] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e85808a4d14a84b245240acc784d51 (127.18.235.131:33801), 005c837d15484b489c90705841b7376b (127.18.235.129:44433)
I20260927 18:46:07.338806 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.335641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:44433 (local address 127.18.235.130:43261)
0927 18:46:07.335767 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.335783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.335909 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.336150 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.336153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.336170 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:07.336274 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.336280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.336967 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.336970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.337820 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.337829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.338424 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.338428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.338526 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.338544 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.338606 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.338671 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":22}
I20260927 18:46:07.338805 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.335802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:43261 (local address 127.18.235.129:44433)
0927 18:46:07.335912 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.335916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.335931 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:07.336015 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.336017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.336056 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.336124 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:07.336388 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.336867 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.337939 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.338320 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.338419 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.338469 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.338629 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.338695 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:46:07.338949 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.335783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:35305 (local address 127.18.235.131:33801)
0927 18:46:07.335893 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.335898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.336124 (+ 226us) server_negotiation.cc:407] Connection header received
0927 18:46:07.336207 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.336210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.336251 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.336311 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:07.336588 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.337051 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.338098 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.338475 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.338574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.338619 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.338789 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.338852 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:07.338960 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.335641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:33801 (local address 127.18.235.130:35305)
0927 18:46:07.336037 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.336047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.336128 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.336349 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.336352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.336368 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:07.336482 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.336487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.337183 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.337187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.337979 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.337987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.338588 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.338592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.338684 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.338702 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.338762 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.338823 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":129,"thread_start_us":101,"threads_started":1}
I20260927 18:46:07.339536 6497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a649f40254894951b84d9b9a388d0" candidate_uuid: "b37472d8233d4a959b39f5461538dffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e85808a4d14a84b245240acc784d51" is_pre_election: true
I20260927 18:46:07.339807 6497 raft_consensus.cc:2454] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37472d8233d4a959b39f5461538dffe in term 0.
I20260927 18:46:07.339922 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a649f40254894951b84d9b9a388d0" candidate_uuid: "b37472d8233d4a959b39f5461538dffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "005c837d15484b489c90705841b7376b" is_pre_election: true
I20260927 18:46:07.340169 6229 raft_consensus.cc:2454] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37472d8233d4a959b39f5461538dffe in term 0.
I20260927 18:46:07.340267 6297 leader_election.cc:302] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b37472d8233d4a959b39f5461538dffe, f9e85808a4d14a84b245240acc784d51; no voters:
I20260927 18:46:07.340516 6566 raft_consensus.cc:2781] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:07.340595 6566 raft_consensus.cc:482] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:07.340624 6566 raft_consensus.cc:3037] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:07.341651 6566 raft_consensus.cc:504] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.341964 6566 leader_election.cc:288] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [CANDIDATE]: Term 1 election: Requested vote from peers f9e85808a4d14a84b245240acc784d51 (127.18.235.131:33801), 005c837d15484b489c90705841b7376b (127.18.235.129:44433)
I20260927 18:46:07.342293 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a649f40254894951b84d9b9a388d0" candidate_uuid: "b37472d8233d4a959b39f5461538dffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "005c837d15484b489c90705841b7376b"
I20260927 18:46:07.342312 6497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4a649f40254894951b84d9b9a388d0" candidate_uuid: "b37472d8233d4a959b39f5461538dffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e85808a4d14a84b245240acc784d51"
I20260927 18:46:07.342397 6229 raft_consensus.cc:3037] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:07.342435 6497 raft_consensus.cc:3037] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:07.343346 6229 raft_consensus.cc:2454] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37472d8233d4a959b39f5461538dffe in term 1.
I20260927 18:46:07.343494 6497 raft_consensus.cc:2454] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37472d8233d4a959b39f5461538dffe in term 1.
I20260927 18:46:07.343621 6299 leader_election.cc:302] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005c837d15484b489c90705841b7376b, b37472d8233d4a959b39f5461538dffe; no voters:
I20260927 18:46:07.343775 6566 raft_consensus.cc:2781] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:07.343951 6566 raft_consensus.cc:686] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 1 LEADER]: Becoming Leader. State: Replica: b37472d8233d4a959b39f5461538dffe, State: Running, Role: LEADER
I20260927 18:46:07.344260 6566 consensus_queue.cc:237] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } }
I20260927 18:46:07.347139 6084 catalog_manager.cc:5495] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe reported cstate change: term changed from 0 to 1, leader changed from <none> to b37472d8233d4a959b39f5461538dffe (127.18.235.130). New cstate: current_term: 1 leader_uuid: "b37472d8233d4a959b39f5461538dffe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b37472d8233d4a959b39f5461538dffe" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 39243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:07.357023 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.353940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.18.235.130:39243)
0927 18:46:07.354038 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.354042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.354052 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:46:07.354201 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.354204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.354249 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.354328 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:07.354539 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.355002 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.355788 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.355986 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.356064 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.356327 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:07.356678 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:07.356774 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.356849 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.356898 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:07.361945 6229 raft_consensus.cc:1261] T 7b4a649f40254894951b84d9b9a388d0 P 005c837d15484b489c90705841b7376b [term 1 FOLLOWER]: Refusing update from remote peer b37472d8233d4a959b39f5461538dffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:07.361945 6497 raft_consensus.cc:1261] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 1 FOLLOWER]: Refusing update from remote peer b37472d8233d4a959b39f5461538dffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:07.362519 6570 consensus_queue.cc:1035] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e85808a4d14a84b245240acc784d51" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:07.362663 6566 consensus_queue.cc:1035] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "005c837d15484b489c90705841b7376b" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:07.369547 6575 mvcc.cc:205] Tried to move back new op lower bound from 7334030407108620288 to 7334030407044669440. Current Snapshot: MvccSnapshot[applied={T|T < 7334030407108620288}]
I20260927 18:46:07.370012 6574 mvcc.cc:205] Tried to move back new op lower bound from 7334030407108620288 to 7334030407044669440. Current Snapshot: MvccSnapshot[applied={T|T < 7334030407108620288}]
I20260927 18:46:07.376746 6083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:07.376864 6083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:07.378903 6083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c3db74e600446f28757a748d744df06: Sending DeleteTablet for 3 replicas of tablet 7b4a649f40254894951b84d9b9a388d0
I20260927 18:46:07.379523 6477 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4a649f40254894951b84d9b9a388d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:07 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:53346
I20260927 18:46:07.379535 6343 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4a649f40254894951b84d9b9a388d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:07 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:47008
I20260927 18:46:07.379660 6209 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4a649f40254894951b84d9b9a388d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:07 UTC) from {username='kudu', principal='kudu/127.18.235.190@KRBTEST.COM'} at 127.0.0.1:51568
I20260927 18:46:07.379947 6583 tablet_replica.cc:330] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: stopping tablet replica
I20260927 18:46:07.380228 6583 raft_consensus.cc:2229] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:07.380478 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6146
I20260927 18:46:07.380489 6583 raft_consensus.cc:2258] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:07.381199 6584 tablet_replica.cc:330] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: stopping tablet replica
I20260927 18:46:07.381434 6584 raft_consensus.cc:2229] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:07.381752 6584 raft_consensus.cc:2258] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:07.383011 6584 ts_tablet_manager.cc:1905] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:07.385536 6584 ts_tablet_manager.cc:1918] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:07.385684 6584 log.cc:1196] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/7b4a649f40254894951b84d9b9a388d0
I20260927 18:46:07.386044 6584 ts_tablet_manager.cc:1939] T 7b4a649f40254894951b84d9b9a388d0 P b37472d8233d4a959b39f5461538dffe: Deleting consensus metadata
I20260927 18:46:07.386252 6583 ts_tablet_manager.cc:1905] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:46:07.386794 6071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:44433 (error 108)
I20260927 18:46:07.387097 6068 catalog_manager.cc:4841] TS b37472d8233d4a959b39f5461538dffe (127.18.235.130:39243): tablet 7b4a649f40254894951b84d9b9a388d0 (table test-table [id=820a03ea492b4de9903c986a8b75f105]) successfully deleted
I20260927 18:46:07.387550 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6280
I20260927 18:46:07.389037 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.388766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:44433 (local address 127.0.0.1:51582)
0927 18:46:07.388908 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.388966 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 18:46:07.389307 6071 catalog_manager.cc:4584] TS 005c837d15484b489c90705841b7376b (127.18.235.129:44433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b4a649f40254894951b84d9b9a388d0: Network error: Client connection negotiation failed: client connection to 127.18.235.129:44433: connect: Connection refused (error 111)
I20260927 18:46:07.389420 6583 ts_tablet_manager.cc:1918] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:07.389490 6583 log.cc:1196] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/7b4a649f40254894951b84d9b9a388d0
I20260927 18:46:07.389833 6583 ts_tablet_manager.cc:1939] T 7b4a649f40254894951b84d9b9a388d0 P f9e85808a4d14a84b245240acc784d51: Deleting consensus metadata
I20260927 18:46:07.390792 6069 catalog_manager.cc:4841] TS f9e85808a4d14a84b245240acc784d51 (127.18.235.131:33801): tablet 7b4a649f40254894951b84d9b9a388d0 (table test-table [id=820a03ea492b4de9903c986a8b75f105]) successfully deleted
I20260927 18:46:07.393013 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6414
I20260927 18:46:07.397137 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6052
2026-09-27T18:46:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2070 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T18:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:07Z Disabled control of system clock
I20260927 18:46:07.424079 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45501
--webserver_interface=127.18.235.190
--webserver_port=0
--builtin_ntp_servers=127.18.235.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:46:07.524811 6590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:07.525014 6590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:07.525045 6590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:07.528837 6590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:07.528879 6590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:07.528892 6590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:07.528911 6590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:07.528928 6590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:07.533353 6590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:45501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:07.534109 6590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:07.535209 6590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:07.542439 6590 server_base.cc:1034] running on GCE node
W20260927 18:46:07.542369 6595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.542361 6596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.542369 6598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.543119 6590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:07.544044 6590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:07.545158 6590 hybrid_clock.cc:648] HybridClock initialized: now 1790534767545140 us; error 29 us; skew 500 ppm
I20260927 18:46:07.546732 6590 webserver.cc:460] Webserver started at http://127.18.235.190:42687/ using document root <none> and password file <none>
I20260927 18:46:07.547289 6590 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:07.547366 6590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:07.547545 6590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:07.549381 6590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "2d931f3683cf42d0b548b58cd532b88a"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "942b6b858b9315625fdd9b8f74042576"
server_key_iv: "bc374bcd3159616d299a42dd05932f42"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.549870 6590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "2d931f3683cf42d0b548b58cd532b88a"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "942b6b858b9315625fdd9b8f74042576"
server_key_iv: "bc374bcd3159616d299a42dd05932f42"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.552994 6590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:46:07.555088 6604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.556146 6590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:07.556248 6590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "2d931f3683cf42d0b548b58cd532b88a"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "942b6b858b9315625fdd9b8f74042576"
server_key_iv: "bc374bcd3159616d299a42dd05932f42"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.556340 6590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:07.583364 6590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:07.583904 6590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:07.584049 6590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:07.590023 6590 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:45501
I20260927 18:46:07.590047 6656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:45501 every 8 connection(s)
I20260927 18:46:07.590811 6590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:07.593642 6657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:07.598477 6657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a: Bootstrap starting.
I20260927 18:46:07.598537 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6590
I20260927 18:46:07.598652 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:07.598882 19374 external_mini_cluster.cc:1428] Setting key be0141afa1b93f4875f7b1a55e2e0f5c
I20260927 18:46:07.600742 6657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:07.601437 6657 log.cc:824] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:07.603031 6657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a: No bootstrap required, opened a new log
I20260927 18:46:07.606201 6657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d931f3683cf42d0b548b58cd532b88a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45501 } }
I20260927 18:46:07.606444 6657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:07.606503 6657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d931f3683cf42d0b548b58cd532b88a, State: Initialized, Role: FOLLOWER
I20260927 18:46:07.606887 6657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d931f3683cf42d0b548b58cd532b88a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45501 } }
I20260927 18:46:07.607009 6657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:07.607070 6657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:07.607075 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.600043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.18.235.190:45501)
0927 18:46:07.600594 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.600607 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.600638 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:46:07.601290 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.601308 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.601679 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.601970 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:07.602351 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.603305 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.604146 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.604594 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.604745 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.605491 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:07.605495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:07.605507 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:07.605849 (+ 342us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:07.605899 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:07.605918 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:07.605999 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.606130 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.606359 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260927 18:46:07.607183 6657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:07.607911 6657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d931f3683cf42d0b548b58cd532b88a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45501 } }
I20260927 18:46:07.608242 6657 leader_election.cc:302] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d931f3683cf42d0b548b58cd532b88a; no voters:
I20260927 18:46:07.608530 6657 leader_election.cc:288] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:07.608603 6662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:07.608791 6662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [term 1 LEADER]: Becoming Leader. State: Replica: 2d931f3683cf42d0b548b58cd532b88a, State: Running, Role: LEADER
I20260927 18:46:07.609122 6662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d931f3683cf42d0b548b58cd532b88a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45501 } }
I20260927 18:46:07.609615 6657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:07.610065 6662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d931f3683cf42d0b548b58cd532b88a. Latest consensus state: current_term: 1 leader_uuid: "2d931f3683cf42d0b548b58cd532b88a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d931f3683cf42d0b548b58cd532b88a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45501 } } }
I20260927 18:46:07.610176 6662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:07.610518 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d931f3683cf42d0b548b58cd532b88a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d931f3683cf42d0b548b58cd532b88a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 45501 } } }
I20260927 18:46:07.610637 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:07.611663 6667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:07.614048 6667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:07.619220 6667 catalog_manager.cc:1347] Generated new cluster ID: 92944d00135f486eb5184355a667c3ba
I20260927 18:46:07.619273 6667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:07.661696 6667 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:07.662540 6667 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:07.666401 6667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a: Generated new TSK 0
I20260927 18:46:07.667073 6667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:07.679494 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.129:0
--local_ip_for_outbound_sockets=127.18.235.129
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45501
--builtin_ntp_servers=127.18.235.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:46:07.782327 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:07.782534 6681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:07.782581 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:07.786468 6681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:07.786511 6681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:07.786582 6681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.129
I20260927 18:46:07.791005 6681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.235.129
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.129
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:07.791834 6681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:07.792977 6681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:07.801326 6687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.801326 6686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.801378 6681 server_base.cc:1034] running on GCE node
W20260927 18:46:07.801476 6689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.801865 6681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:07.802423 6681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:07.803562 6681 hybrid_clock.cc:648] HybridClock initialized: now 1790534767803538 us; error 39 us; skew 500 ppm
I20260927 18:46:07.805159 6681 webserver.cc:460] Webserver started at http://127.18.235.129:38881/ using document root <none> and password file <none>
I20260927 18:46:07.805644 6681 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:07.805704 6681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:07.805868 6681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:07.807657 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data/instance:
uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "1b21817bcc15464c0cc277efe1f3a47a"
server_key_iv: "332eec57bc25000171d2cb44d09ff3aa"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.808143 6681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance:
uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "1b21817bcc15464c0cc277efe1f3a47a"
server_key_iv: "332eec57bc25000171d2cb44d09ff3aa"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.811564 6681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:46:07.813755 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.814842 6681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:07.814939 6681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal
uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "1b21817bcc15464c0cc277efe1f3a47a"
server_key_iv: "332eec57bc25000171d2cb44d09ff3aa"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.815022 6681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:07.832556 6681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:07.833074 6681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:07.833266 6681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:07.833750 6681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:07.834569 6681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:07.834615 6681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.834663 6681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:07.834694 6681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.842895 6681 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.129:38547
I20260927 18:46:07.842931 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.129:38547 every 8 connection(s)
I20260927 18:46:07.843647 6681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:07.844607 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6681
I20260927 18:46:07.844710 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:07.844959 19374 external_mini_cluster.cc:1428] Setting key 310bab51e63f6c6626e85dc5cbd98e50
I20260927 18:46:07.846951 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.130:0
--local_ip_for_outbound_sockets=127.18.235.130
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45501
--builtin_ntp_servers=127.18.235.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:46:07.852527 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.845218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.129:40053 (local address 127.18.235.190:45501)
0927 18:46:07.845347 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.845351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.846446 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 18:46:07.848092 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.848095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.848136 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.848212 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:07.849065 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.849665 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.850802 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.851023 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.851118 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.852106 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:07.852110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:07.852113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:07.852126 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:07.852146 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:07.852154 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:07.852210 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.852339 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.852394 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:46:07.853300 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.845622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45501 (local address 127.18.235.129:40053)
0927 18:46:07.846313 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.846361 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.847890 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.848383 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.848392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.848493 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:07.848941 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.848953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.849783 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.849786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.850688 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.850698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.851149 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.851153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.851311 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.851869 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:07.851870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:07.851905 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:07.851967 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:07.852219 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:07.852227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:07.852233 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.852333 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.852577 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
I20260927 18:46:07.854332 6809 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45501
I20260927 18:46:07.854526 6809 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:07.854950 6809 heartbeater.cc:511] Master 127.18.235.190:45501 requested a full tablet report, sending...
I20260927 18:46:07.856416 6621 ts_manager.cc:194] Registered new tserver with Master: f9d5bd0a3a8e4f03b4d8d419168275f8 (127.18.235.129:38547)
I20260927 18:46:07.857939 6621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.129:40053
W20260927 18:46:07.949048 6812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:07.949290 6812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:07.949324 6812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:07.953154 6812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:07.953197 6812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:07.953248 6812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.130
I20260927 18:46:07.957690 6812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.235.130
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.130
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:07.958474 6812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:07.959689 6812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:07.968247 6817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.968253 6818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.968253 6820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.968480 6812 server_base.cc:1034] running on GCE node
I20260927 18:46:07.968896 6812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:07.969499 6812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:07.970643 6812 hybrid_clock.cc:648] HybridClock initialized: now 1790534767970628 us; error 28 us; skew 500 ppm
I20260927 18:46:07.972246 6812 webserver.cc:460] Webserver started at http://127.18.235.130:37555/ using document root <none> and password file <none>
I20260927 18:46:07.972738 6812 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:07.972795 6812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:07.972940 6812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:07.974766 6812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data/instance:
uuid: "dd456bf7dbd441989f345c854bbab35e"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "22e1232471050dedbc4933e77ebe2f66"
server_key_iv: "217d3f6098fbd1aacb56c40835bb313e"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.975235 6812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance:
uuid: "dd456bf7dbd441989f345c854bbab35e"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "22e1232471050dedbc4933e77ebe2f66"
server_key_iv: "217d3f6098fbd1aacb56c40835bb313e"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.978701 6812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:46:07.980911 6826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.982036 6812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:46:07.982146 6812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal
uuid: "dd456bf7dbd441989f345c854bbab35e"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-6lr6"
server_key: "22e1232471050dedbc4933e77ebe2f66"
server_key_iv: "217d3f6098fbd1aacb56c40835bb313e"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.982241 6812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:07.995954 6812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:07.996503 6812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:07.996671 6812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:07.997200 6812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:07.998100 6812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:07.998152 6812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.998205 6812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:07.998237 6812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:08.006635 6812 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.130:38699
I20260927 18:46:08.006661 6939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.130:38699 every 8 connection(s)
I20260927 18:46:08.007409 6812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:08.012974 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6812
I20260927 18:46:08.013095 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:08.013437 19374 external_mini_cluster.cc:1428] Setting key 08cb090e5b2f27c7966319cd5494054c
I20260927 18:46:08.015381 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.235.131:0
--local_ip_for_outbound_sockets=127.18.235.131
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45501
--builtin_ntp_servers=127.18.235.148:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:46:08.016242 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.009012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.130:44189 (local address 127.18.235.190:45501)
0927 18:46:08.009167 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.009171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.010168 (+ 997us) server_negotiation.cc:407] Connection header received
0927 18:46:08.011487 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.011490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.011539 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.011621 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.012444 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.013050 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.014365 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.014657 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.014758 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.015834 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.015837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.015839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:08.015851 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.015874 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:08.015883 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.015940 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.016079 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.016129 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:08.017197 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.009382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45501 (local address 127.18.235.130:44189)
0927 18:46:08.010033 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.010068 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.011311 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.011780 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.011786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.011872 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:08.012309 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.012320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.013213 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.013216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.014256 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.014265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.014800 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.014805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.014985 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.015552 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:08.015554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:08.015599 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:08.015675 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:08.015951 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:08.015958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:08.015963 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.016066 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.016312 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260927 18:46:08.018252 6940 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45501
I20260927 18:46:08.018462 6940 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:08.018963 6940 heartbeater.cc:511] Master 127.18.235.190:45501 requested a full tablet report, sending...
I20260927 18:46:08.019977 6621 ts_manager.cc:194] Registered new tserver with Master: dd456bf7dbd441989f345c854bbab35e (127.18.235.130:38699)
I20260927 18:46:08.020509 6621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.130:44189
W20260927 18:46:08.117755 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:08.117967 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:08.118016 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:08.121868 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:08.121912 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:08.121980 6943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.235.131
I20260927 18:46:08.126446 6943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.235.131
--webserver_port=0
--tserver_master_addrs=127.18.235.190:45501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.235.131
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:08.127291 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:08.128455 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:08.136608 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:08.136631 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:08.136667 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:08.136833 6943 server_base.cc:1034] running on GCE node
I20260927 18:46:08.137240 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:08.137775 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:08.138974 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790534768138959 us; error 33 us; skew 500 ppm
I20260927 18:46:08.140599 6943 webserver.cc:460] Webserver started at http://127.18.235.131:34021/ using document root <none> and password file <none>
I20260927 18:46:08.141078 6943 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:08.141172 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:08.141366 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:08.143141 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data/instance:
uuid: "7316e251c31245a98046714aaf775c0f"
format_stamp: "Formatted at 2026-09-27 18:46:08 on dist-test-slave-6lr6"
server_key: "f6c8804185c30477fb1bd0449af00e5b"
server_key_iv: "6a22a0dbd9959d8617994b447b08d586"
server_key_version: "encryptionkey@0"
I20260927 18:46:08.143636 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance:
uuid: "7316e251c31245a98046714aaf775c0f"
format_stamp: "Formatted at 2026-09-27 18:46:08 on dist-test-slave-6lr6"
server_key: "f6c8804185c30477fb1bd0449af00e5b"
server_key_iv: "6a22a0dbd9959d8617994b447b08d586"
server_key_version: "encryptionkey@0"
I20260927 18:46:08.146826 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:08.148993 6957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:08.150058 6943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:08.150153 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal
uuid: "7316e251c31245a98046714aaf775c0f"
format_stamp: "Formatted at 2026-09-27 18:46:08 on dist-test-slave-6lr6"
server_key: "f6c8804185c30477fb1bd0449af00e5b"
server_key_iv: "6a22a0dbd9959d8617994b447b08d586"
server_key_version: "encryptionkey@0"
I20260927 18:46:08.150242 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:08.159286 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:08.159811 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:08.159952 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:08.160456 6943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:08.161453 6943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:08.161506 6943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:08.161552 6943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:08.161579 6943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:08.171307 6943 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.131:39757
I20260927 18:46:08.171367 7070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.131:39757 every 8 connection(s)
I20260927 18:46:08.172127 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:08.179853 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.173627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:51339 (local address 127.18.235.190:45501)
0927 18:46:08.173752 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.173756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.174710 (+ 954us) server_negotiation.cc:407] Connection header received
0927 18:46:08.175929 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.175931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.175973 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.176041 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.176831 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.177382 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.178388 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.178588 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.178650 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.179529 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.179531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.179533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:08.179542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.179556 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:08.179562 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.179606 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.179708 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.179748 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:08.180547 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.173990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:45501 (local address 127.18.235.131:51339)
0927 18:46:08.174558 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.174619 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.175771 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.176178 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.176184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.176268 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:08.176729 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.176741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.177470 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.177473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.178295 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.178303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.178685 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.178688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.178812 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.179325 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:08.179327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:08.179365 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:08.179420 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:08.179618 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:08.179622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:08.179626 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.179703 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.179891 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260927 18:46:08.181020 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 6943
I20260927 18:46:08.181146 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:08.181433 19374 external_mini_cluster.cc:1428] Setting key dce2aa6bafe92e5dd131fa6eb0da2471
I20260927 18:46:08.181607 7071 heartbeater.cc:347] Connected to a master server at 127.18.235.190:45501
I20260927 18:46:08.181792 7071 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:08.182294 7071 heartbeater.cc:511] Master 127.18.235.190:45501 requested a full tablet report, sending...
I20260927 18:46:08.183240 6620 ts_manager.cc:194] Registered new tserver with Master: 7316e251c31245a98046714aaf775c0f (127.18.235.131:39757)
I20260927 18:46:08.183779 6620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.235.131:51339
I20260927 18:46:08.194676 19374 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:08.199793 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.196401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.18.235.190:45501)
0927 18:46:08.196514 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.196518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.196678 (+ 160us) server_negotiation.cc:407] Connection header received
0927 18:46:08.196791 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.196794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.196834 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.196896 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.197161 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.197650 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.198459 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.198664 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.198753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.199416 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.199419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.199421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:08.199431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.199449 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:08.199457 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.199513 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.199601 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.199654 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:46:08.202255 6620 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:08.205077 6620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:08.218742 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.212676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:38547 (local address 127.0.0.1:33322)
0927 18:46:08.213377 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.213423 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.213529 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.214388 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.214393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.214413 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.214530 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.214538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.215944 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.215947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.217283 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.217293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.218242 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.218245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.218344 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.218382 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.218498 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.218607 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":109,"thread_start_us":55,"threads_started":1}
I20260927 18:46:08.219002 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.213220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.131:39757 (local address 127.0.0.1:50298)
0927 18:46:08.214538 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.214550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.214636 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.215055 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.215058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.215073 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.215146 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.215152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.216050 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.216053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.217512 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.217524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.218507 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.218510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.218599 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.218622 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.218680 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.218744 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":80,"spinlock_wait_cycles":3840,"thread_start_us":88,"threads_started":1}
I20260927 18:46:08.219067 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.213594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.18.235.131:39757)
0927 18:46:08.214116 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.214128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.214723 (+ 595us) server_negotiation.cc:407] Connection header received
0927 18:46:08.214797 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.214802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.214943 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.215058 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.215244 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.215952 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.217620 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.218413 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.218519 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.218733 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.218780 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.218926 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":134,"threads_started":1}
I20260927 18:46:08.219112 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.213375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.18.235.130:38699)
0927 18:46:08.213787 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.213799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.214203 (+ 404us) server_negotiation.cc:407] Connection header received
0927 18:46:08.214271 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.214275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.214450 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.214603 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.214794 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.215859 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.217646 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.218241 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.218348 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.218503 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.218878 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.218971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":119,"threads_started":1}
I20260927 18:46:08.219393 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.212803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38699 (local address 127.0.0.1:35022)
0927 18:46:08.213895 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.213907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.214001 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.214600 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.214603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.214626 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.214711 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.214717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.215970 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.215974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.217532 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.217539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.218559 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.218563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.218641 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.218662 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.218736 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.218796 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":145,"spinlock_wait_cycles":384,"thread_start_us":92,"threads_started":1}
I20260927 18:46:08.219658 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.212808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.18.235.129:38547)
0927 18:46:08.213976 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.213990 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.214013 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:46:08.214100 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.214104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.214256 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.214367 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.214658 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.215827 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.217395 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.218133 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.218249 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.218416 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.219409 (+ 993us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.219511 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":99,"threads_started":1}
I20260927 18:46:08.221999 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc1855b66e9465685c92a234027ccba (DEFAULT_TABLE table=test-table [id=3fb52f646b784b378022b8121f17b8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:08.222131 6874 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc1855b66e9465685c92a234027ccba (DEFAULT_TABLE table=test-table [id=3fb52f646b784b378022b8121f17b8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:08.222132 6743 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc1855b66e9465685c92a234027ccba (DEFAULT_TABLE table=test-table [id=3fb52f646b784b378022b8121f17b8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:08.222890 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc1855b66e9465685c92a234027ccba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:08.223088 6874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc1855b66e9465685c92a234027ccba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:08.223097 6743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc1855b66e9465685c92a234027ccba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:08.228190 7088 tablet_bootstrap.cc:492] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Bootstrap starting.
I20260927 18:46:08.229560 7089 tablet_bootstrap.cc:492] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: Bootstrap starting.
I20260927 18:46:08.230458 7088 tablet_bootstrap.cc:654] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:08.231317 7088 log.cc:824] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:08.231874 7089 tablet_bootstrap.cc:654] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:08.231985 7090 tablet_bootstrap.cc:492] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Bootstrap starting.
I20260927 18:46:08.232668 7089 log.cc:824] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:08.232952 7088 tablet_bootstrap.cc:492] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: No bootstrap required, opened a new log
I20260927 18:46:08.233150 7088 ts_tablet_manager.cc:1397] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 18:46:08.233942 7090 tablet_bootstrap.cc:654] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:08.234543 7090 log.cc:824] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:08.234596 7089 tablet_bootstrap.cc:492] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: No bootstrap required, opened a new log
I20260927 18:46:08.234757 7089 ts_tablet_manager.cc:1397] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:46:08.236093 7090 tablet_bootstrap.cc:492] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: No bootstrap required, opened a new log
I20260927 18:46:08.236253 7090 ts_tablet_manager.cc:1397] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:46:08.237196 7088 raft_consensus.cc:348] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.237442 7088 raft_consensus.cc:374] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:08.237495 7088 raft_consensus.cc:729] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7316e251c31245a98046714aaf775c0f, State: Initialized, Role: FOLLOWER
I20260927 18:46:08.237921 7088 consensus_queue.cc:260] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.238545 7071 heartbeater.cc:503] Master 127.18.235.190:45501 was elected leader, sending a full tablet report...
I20260927 18:46:08.238618 7089 raft_consensus.cc:348] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.238844 7089 raft_consensus.cc:374] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:08.238897 7089 raft_consensus.cc:729] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d5bd0a3a8e4f03b4d8d419168275f8, State: Initialized, Role: FOLLOWER
I20260927 18:46:08.239303 7089 consensus_queue.cc:260] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.239439 7090 raft_consensus.cc:348] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.239669 7090 raft_consensus.cc:374] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:08.239740 7090 raft_consensus.cc:729] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd456bf7dbd441989f345c854bbab35e, State: Initialized, Role: FOLLOWER
I20260927 18:46:08.239848 7088 ts_tablet_manager.cc:1428] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:46:08.240137 7090 consensus_queue.cc:260] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.240302 6809 heartbeater.cc:503] Master 127.18.235.190:45501 was elected leader, sending a full tablet report...
I20260927 18:46:08.240721 6940 heartbeater.cc:503] Master 127.18.235.190:45501 was elected leader, sending a full tablet report...
I20260927 18:46:08.240166 7089 ts_tablet_manager.cc:1428] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:08.241048 7090 ts_tablet_manager.cc:1428] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:46:08.243054 7094 raft_consensus.cc:482] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:08.243233 7094 raft_consensus.cc:504] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.244246 7094 leader_election.cc:288] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d5bd0a3a8e4f03b4d8d419168275f8 (127.18.235.129:38547), dd456bf7dbd441989f345c854bbab35e (127.18.235.130:38699)
I20260927 18:46:08.247210 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.244267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:42885 (local address 127.18.235.129:38547)
0927 18:46:08.244365 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.244370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.244437 (+ 67us) server_negotiation.cc:407] Connection header received
0927 18:46:08.244531 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.244533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.244567 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.244636 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.244864 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.245409 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.246367 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.246742 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.246837 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.246890 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.247046 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.247108 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:46:08.247211 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.244222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:38547 (local address 127.18.235.131:42885)
0927 18:46:08.244365 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.244377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.244461 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.244660 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.244663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.244678 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.244775 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.244780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.245501 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.245505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.246254 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.246263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.246837 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.246841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.246938 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.246960 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.247024 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.247100 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 18:46:08.247614 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.244274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.235.131:50833 (local address 127.18.235.130:38699)
0927 18:46:08.244366 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.244370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.244839 (+ 469us) server_negotiation.cc:407] Connection header received
0927 18:46:08.244940 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.244943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.244980 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.245037 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.245308 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.245770 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.246731 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.247095 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.247190 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.247231 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.247445 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.247513 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:46:08.247748 6763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc1855b66e9465685c92a234027ccba" candidate_uuid: "7316e251c31245a98046714aaf775c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" is_pre_election: true
I20260927 18:46:08.247912 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.244222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.130:38699 (local address 127.18.235.131:50833)
0927 18:46:08.244765 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.244777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.244871 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.245053 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.245055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.245068 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.245215 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.245221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.245862 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.245865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.246617 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.246625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.247207 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.247211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.247306 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.247327 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.247722 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.247794 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":301,"thread_start_us":69,"threads_started":1}
I20260927 18:46:08.247965 6763 raft_consensus.cc:2454] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7316e251c31245a98046714aaf775c0f in term 0.
I20260927 18:46:08.248381 6961 leader_election.cc:302] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7316e251c31245a98046714aaf775c0f, f9d5bd0a3a8e4f03b4d8d419168275f8; no voters:
I20260927 18:46:08.248581 7094 raft_consensus.cc:2781] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:08.248672 7094 raft_consensus.cc:482] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:08.248718 7094 raft_consensus.cc:3037] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:08.248796 6894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc1855b66e9465685c92a234027ccba" candidate_uuid: "7316e251c31245a98046714aaf775c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd456bf7dbd441989f345c854bbab35e" is_pre_election: true
I20260927 18:46:08.249017 6894 raft_consensus.cc:2454] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7316e251c31245a98046714aaf775c0f in term 0.
I20260927 18:46:08.249673 7094 raft_consensus.cc:504] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.249945 7094 leader_election.cc:288] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [CANDIDATE]: Term 1 election: Requested vote from peers f9d5bd0a3a8e4f03b4d8d419168275f8 (127.18.235.129:38547), dd456bf7dbd441989f345c854bbab35e (127.18.235.130:38699)
I20260927 18:46:08.250223 6894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc1855b66e9465685c92a234027ccba" candidate_uuid: "7316e251c31245a98046714aaf775c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd456bf7dbd441989f345c854bbab35e"
I20260927 18:46:08.250255 6763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc1855b66e9465685c92a234027ccba" candidate_uuid: "7316e251c31245a98046714aaf775c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8"
I20260927 18:46:08.250331 6894 raft_consensus.cc:3037] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:08.250340 6763 raft_consensus.cc:3037] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:08.251364 6894 raft_consensus.cc:2454] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7316e251c31245a98046714aaf775c0f in term 1.
I20260927 18:46:08.251364 6763 raft_consensus.cc:2454] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7316e251c31245a98046714aaf775c0f in term 1.
I20260927 18:46:08.251618 6961 leader_election.cc:302] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7316e251c31245a98046714aaf775c0f, f9d5bd0a3a8e4f03b4d8d419168275f8; no voters:
I20260927 18:46:08.251801 7094 raft_consensus.cc:2781] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:08.251945 7094 raft_consensus.cc:686] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 1 LEADER]: Becoming Leader. State: Replica: 7316e251c31245a98046714aaf775c0f, State: Running, Role: LEADER
I20260927 18:46:08.252188 7094 consensus_queue.cc:237] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } }
I20260927 18:46:08.254783 6619 catalog_manager.cc:5495] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7316e251c31245a98046714aaf775c0f (127.18.235.131). New cstate: current_term: 1 leader_uuid: "7316e251c31245a98046714aaf775c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7316e251c31245a98046714aaf775c0f" member_type: VOTER last_known_addr { host: "127.18.235.131" port: 39757 } health_report { overall_health: HEALTHY } } }
W20260927 18:46:08.258227 6941 tablet.cc:2369] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:08.273540 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.270722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.18.235.131:39757)
0927 18:46:08.270816 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.270819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.270830 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:46:08.270948 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.270950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.270993 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.271067 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:08.271225 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.271702 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.272453 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.272659 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.272724 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.272921 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:08.273278 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:08.273355 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.273401 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.273438 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 18:46:08.278316 6894 raft_consensus.cc:1261] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 1 FOLLOWER]: Refusing update from remote peer 7316e251c31245a98046714aaf775c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:08.278316 6763 raft_consensus.cc:1261] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 1 FOLLOWER]: Refusing update from remote peer 7316e251c31245a98046714aaf775c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:08.278862 7098 consensus_queue.cc:1035] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d5bd0a3a8e4f03b4d8d419168275f8" member_type: VOTER last_known_addr { host: "127.18.235.129" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:08.279021 7094 consensus_queue.cc:1035] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd456bf7dbd441989f345c854bbab35e" member_type: VOTER last_known_addr { host: "127.18.235.130" port: 38699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:08.287712 7102 mvcc.cc:205] Tried to move back new op lower bound from 7334030410862321664 to 7334030410763456512. Current Snapshot: MvccSnapshot[applied={T|T < 7334030410862321664}]
I20260927 18:46:08.292874 6621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:08.293022 6621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:08.295189 6621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d931f3683cf42d0b548b58cd532b88a: Sending DeleteTablet for 3 replicas of tablet 1bc1855b66e9465685c92a234027ccba
I20260927 18:46:08.295840 6874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc1855b66e9465685c92a234027ccba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:08 UTC) from {username='slave'} at 127.0.0.1:35022
I20260927 18:46:08.295841 6743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc1855b66e9465685c92a234027ccba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:08 UTC) from {username='slave'} at 127.0.0.1:33322
I20260927 18:46:08.295992 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc1855b66e9465685c92a234027ccba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:08 UTC) from {username='slave'} at 127.0.0.1:50298
I20260927 18:46:08.296381 7111 tablet_replica.cc:330] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: stopping tablet replica
I20260927 18:46:08.296454 7112 tablet_replica.cc:330] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8: stopping tablet replica
I20260927 18:46:08.296648 7111 raft_consensus.cc:2229] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:08.296717 7112 raft_consensus.cc:2229] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:08.296921 7111 raft_consensus.cc:2258] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:08.296962 7112 raft_consensus.cc:2258] T 1bc1855b66e9465685c92a234027ccba P f9d5bd0a3a8e4f03b4d8d419168275f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:08.297031 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6681
I20260927 18:46:08.297953 7113 tablet_replica.cc:330] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: stopping tablet replica
I20260927 18:46:08.298167 7113 raft_consensus.cc:2229] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:08.298456 7113 raft_consensus.cc:2258] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:08.298934 7111 ts_tablet_manager.cc:1905] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:08.301548 7111 ts_tablet_manager.cc:1918] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:08.301653 7111 log.cc:1196] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-1/wal/wals/1bc1855b66e9465685c92a234027ccba
I20260927 18:46:08.301942 7111 ts_tablet_manager.cc:1939] T 1bc1855b66e9465685c92a234027ccba P dd456bf7dbd441989f345c854bbab35e: Deleting consensus metadata
I20260927 18:46:08.303048 6607 catalog_manager.cc:4841] TS dd456bf7dbd441989f345c854bbab35e (127.18.235.130:38699): tablet 1bc1855b66e9465685c92a234027ccba (table test-table [id=3fb52f646b784b378022b8121f17b8a6]) successfully deleted
W20260927 18:46:08.303480 6608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.235.129:38547 (error 108)
I20260927 18:46:08.304001 7113 ts_tablet_manager.cc:1905] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:08.304288 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.304056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.129:38547 (local address 127.0.0.1:33324)
0927 18:46:08.304161 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.304222 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.129:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:46:08.304430 6608 catalog_manager.cc:4584] TS f9d5bd0a3a8e4f03b4d8d419168275f8 (127.18.235.129:38547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bc1855b66e9465685c92a234027ccba: Network error: Client connection negotiation failed: client connection to 127.18.235.129:38547: connect: Connection refused (error 111)
I20260927 18:46:08.304674 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6812
I20260927 18:46:08.307719 7113 ts_tablet_manager.cc:1918] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:08.307808 7113 log.cc:1196] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Deleting WAL directory at /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534532155427-19374-0/minicluster-data/ts-2/wal/wals/1bc1855b66e9465685c92a234027ccba
I20260927 18:46:08.308135 7113 ts_tablet_manager.cc:1939] T 1bc1855b66e9465685c92a234027ccba P 7316e251c31245a98046714aaf775c0f: Deleting consensus metadata
I20260927 18:46:08.309120 6606 catalog_manager.cc:4841] TS 7316e251c31245a98046714aaf775c0f (127.18.235.131:39757): tablet 1bc1855b66e9465685c92a234027ccba (table test-table [id=3fb52f646b784b378022b8121f17b8a6]) successfully deleted
I20260927 18:46:08.311245 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6943
I20260927 18:46:08.315459 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 6590
2026-09-27T18:46:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (916 ms)
[----------] 6 tests from EncryptionPolicyTest (9322 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:08 dist-test-slave-6lr6 krb5kdc[7117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:08 dist-test-slave-6lr6 krb5kdc[7117](info): set up 2 sockets
Sep 27 18:46:08 dist-test-slave-6lr6 krb5kdc[7117](info): commencing operation
krb5kdc: starting...
W20260927 18:46:09.260818 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:09 dist-test-slave-6lr6 krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:09.366920 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:09.467841 7130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:09.468055 7130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:09.468096 7130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:09.471917 7130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:09.471963 7130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:09.471980 7130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:09.471994 7130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:09.472013 7130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:09.476820 7130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:09.477640 7130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:09.478471 7130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:09.485849 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:09.485880 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:09.485912 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:09.486037 7130 server_base.cc:1034] running on GCE node
I20260927 18:46:09.486577 7130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:09.487510 7130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:09.488662 7130 hybrid_clock.cc:648] HybridClock initialized: now 1790534769488650 us; error 25 us; skew 500 ppm
Sep 27 18:46:09 dist-test-slave-6lr6 krb5kdc[7117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:09.498767 7130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:09.499539 7130 webserver.cc:460] Webserver started at http://127.0.0.1:44557/ using document root <none> and password file <none>
I20260927 18:46:09.500020 7130 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:09.500082 7130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:09.500273 7130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:09.502225 7130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca"
format_stamp: "Formatted at 2026-09-27 18:46:09 on dist-test-slave-6lr6"
server_key: "d6a4e206d580395fedb4e1cb537407db"
server_key_iv: "cb1064087c622b1bafa9800a82cf0b3d"
server_key_version: "encryptionkey@0"
I20260927 18:46:09.502732 7130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca"
format_stamp: "Formatted at 2026-09-27 18:46:09 on dist-test-slave-6lr6"
server_key: "d6a4e206d580395fedb4e1cb537407db"
server_key_iv: "cb1064087c622b1bafa9800a82cf0b3d"
server_key_version: "encryptionkey@0"
I20260927 18:46:09.506026 7130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:09.508111 7145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:09.509227 7130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:09.509326 7130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca"
format_stamp: "Formatted at 2026-09-27 18:46:09 on dist-test-slave-6lr6"
server_key: "d6a4e206d580395fedb4e1cb537407db"
server_key_iv: "cb1064087c622b1bafa9800a82cf0b3d"
server_key_version: "encryptionkey@0"
I20260927 18:46:09.509405 7130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:09.530802 7130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:09.533471 7130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:09.533629 7130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:09.539714 7130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34213
I20260927 18:46:09.539741 7197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34213 every 8 connection(s)
I20260927 18:46:09.540488 7130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:09.541306 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7130
I20260927 18:46:09.541416 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:09.541671 19374 external_mini_cluster.cc:1428] Setting key fc8ec82cffaa1375c79ecbe1795e2df1
I20260927 18:46:09.543156 7198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:09.548039 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca: Bootstrap starting.
Sep 27 18:46:09 dist-test-slave-6lr6 krb5kdc[7117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:46:09.550297 7198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:09.551116 7198 log.cc:824] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:09.553000 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca: No bootstrap required, opened a new log
I20260927 18:46:09.554512 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:09.542877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43029 (local address 127.0.0.1:34213)
0927 18:46:09.543276 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:09.543293 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:09.543324 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:46:09.543906 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:09.543921 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:09.544216 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:09.544490 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:09.545250 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:09.546189 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:09.546992 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:09.547421 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:09.547553 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:09.550027 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:09.550032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:09.550043 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:09.550067 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:09.551707 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:09.552133 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:09.552139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:09.552144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:09.552280 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:09.552532 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:09.552535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:09.552536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:09.552983 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:09.553214 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:09.553464 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 18:46:09.553671 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260927 18:46:09.556927 7198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260927 18:46:09.557137 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:09.557179 7198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:09.557238 7198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7960ea70a1ee4ecd9494d2a7c8ad2dca, State: Initialized, Role: FOLLOWER
I20260927 18:46:09.557619 7198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260927 18:46:09.557740 7198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:09.557799 7198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:09.557885 7198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:09.558691 7198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260927 18:46:09.559012 7198 leader_election.cc:302] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7960ea70a1ee4ecd9494d2a7c8ad2dca; no voters:
I20260927 18:46:09.559296 7198 leader_election.cc:288] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:09.559747 7209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:09.560050 7209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [term 1 LEADER]: Becoming Leader. State: Replica: 7960ea70a1ee4ecd9494d2a7c8ad2dca, State: Running, Role: LEADER
I20260927 18:46:09.560348 7209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260927 18:46:09.560591 7198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:09.561395 7210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7960ea70a1ee4ecd9494d2a7c8ad2dca. Latest consensus state: current_term: 1 leader_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } } }
I20260927 18:46:09.561575 7210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:09.561738 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ea70a1ee4ecd9494d2a7c8ad2dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } } }
I20260927 18:46:09.561838 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:09.562042 7217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:09.564870 7217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:09.567477 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:09.558861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44709 (local address 127.0.0.1:34213)
0927 18:46:09.558955 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:09.558958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:09.559166 (+ 208us) server_negotiation.cc:407] Connection header received
0927 18:46:09.559310 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:09.559329 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:09.559368 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:09.559430 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:09.560191 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:09.560890 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:09.561985 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:09.562313 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:09.562424 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:09.563953 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:09.563957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:09.563959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:09.563982 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:09.565450 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:09.566226 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:09.566229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:09.566231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:09.566310 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:09.566680 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:09.566683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:09.566685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:09.566865 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:09.566999 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:09.567265 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:46:09.567349 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:46:09.569746 7217 catalog_manager.cc:1347] Generated new cluster ID: 7aef01e38ee142d7a93b1ff9b914aab4
I20260927 18:46:09.569802 7217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:09.604839 7217 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:09.606053 7217 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:09.615736 7217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7960ea70a1ee4ecd9494d2a7c8ad2dca: Generated new TSK 0
I20260927 18:46:09.616464 7217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:09.622653 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7130
W20260927 18:46:09.627326 7115 connection.cc:535] client connection to 127.0.0.1:34213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34213): Connection reset by peer (error 104)
2026-09-27T18:46:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:09 dist-test-slave-6lr6 krb5kdc[7229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:09 dist-test-slave-6lr6 krb5kdc[7229](info): set up 2 sockets
Sep 27 18:46:09 dist-test-slave-6lr6 krb5kdc[7229](info): commencing operation
krb5kdc: starting...
W20260927 18:46:10.564850 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:10 dist-test-slave-6lr6 krb5kdc[7229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:10.673899 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:10.778378 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:10.778617 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:10.778656 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:10.782529 7242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:10.782577 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:10.782595 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:10.782610 7242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:10.782624 7242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:10.787583 7242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:10.788491 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:10.789451 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:10.797165 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:10.797156 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:10.797308 7242 server_base.cc:1034] running on GCE node
W20260927 18:46:10.797156 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:10.797886 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:10.798884 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:10.800037 7242 hybrid_clock.cc:648] HybridClock initialized: now 1790534770800023 us; error 26 us; skew 500 ppm
Sep 27 18:46:10 dist-test-slave-6lr6 krb5kdc[7229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:10.806264 7242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:10.807049 7242 webserver.cc:460] Webserver started at http://127.0.0.1:40713/ using document root <none> and password file <none>
I20260927 18:46:10.807512 7242 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:10.807564 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:10.807727 7242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:10.809640 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "bf867ca5d47e4290a54762b39c8f862c"
format_stamp: "Formatted at 2026-09-27 18:46:10 on dist-test-slave-6lr6"
server_key: "a97b03b7a77993f3fa4ce1ad43534332"
server_key_iv: "3eba35785bfa02a52ed3083124e8c327"
server_key_version: "encryptionkey@0"
I20260927 18:46:10.810112 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "bf867ca5d47e4290a54762b39c8f862c"
format_stamp: "Formatted at 2026-09-27 18:46:10 on dist-test-slave-6lr6"
server_key: "a97b03b7a77993f3fa4ce1ad43534332"
server_key_iv: "3eba35785bfa02a52ed3083124e8c327"
server_key_version: "encryptionkey@0"
I20260927 18:46:10.813452 7242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:10.815583 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:10.816663 7242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:10.816761 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "bf867ca5d47e4290a54762b39c8f862c"
format_stamp: "Formatted at 2026-09-27 18:46:10 on dist-test-slave-6lr6"
server_key: "a97b03b7a77993f3fa4ce1ad43534332"
server_key_iv: "3eba35785bfa02a52ed3083124e8c327"
server_key_version: "encryptionkey@0"
I20260927 18:46:10.816838 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:10.834611 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:10.836966 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:10.837095 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:10.843223 7242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45355
I20260927 18:46:10.843237 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45355 every 8 connection(s)
I20260927 18:46:10.844074 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:10.846606 7310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:10.848773 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7242
I20260927 18:46:10.848877 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:10.849143 19374 external_mini_cluster.cc:1428] Setting key 8351299d8d53b9d9d066cb8769796918
I20260927 18:46:10.851948 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c: Bootstrap starting.
I20260927 18:46:10.854235 7310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:10.854980 7310 log.cc:824] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:10.857035 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c: No bootstrap required, opened a new log
I20260927 18:46:10.857275 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:10.850269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45657 (local address 127.0.0.1:45355)
0927 18:46:10.850617 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:10.850630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:10.850662 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:46:10.851253 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:10.851268 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:10.851578 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:10.851842 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:10.852191 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:10.853098 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:10.854234 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:10.854687 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:10.854807 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:10.855583 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:10.855589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:10.855604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:10.856029 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:10.856079 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:10.856101 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:10.856194 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:10.856374 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 18:46:10.856488 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260927 18:46:10.860137 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:10.860792 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf867ca5d47e4290a54762b39c8f862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260927 18:46:10.861056 7310 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:10.861132 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf867ca5d47e4290a54762b39c8f862c, State: Initialized, Role: FOLLOWER
I20260927 18:46:10.861580 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf867ca5d47e4290a54762b39c8f862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260927 18:46:10.861747 7310 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:10.861814 7310 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:10.861904 7310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:10.862865 7310 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf867ca5d47e4290a54762b39c8f862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260927 18:46:10.863185 7310 leader_election.cc:302] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf867ca5d47e4290a54762b39c8f862c; no voters:
I20260927 18:46:10.863440 7310 leader_election.cc:288] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:10.863512 7320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:10.863698 7320 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [term 1 LEADER]: Becoming Leader. State: Replica: bf867ca5d47e4290a54762b39c8f862c, State: Running, Role: LEADER
I20260927 18:46:10.864012 7320 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf867ca5d47e4290a54762b39c8f862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260927 18:46:10.864562 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:10.865626 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf867ca5d47e4290a54762b39c8f862c. Latest consensus state: current_term: 1 leader_uuid: "bf867ca5d47e4290a54762b39c8f862c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf867ca5d47e4290a54762b39c8f862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } } }
I20260927 18:46:10.865983 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:10.866117 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:10.862208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41151 (local address 127.0.0.1:45355)
0927 18:46:10.862324 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:10.862328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:10.862549 (+ 221us) server_negotiation.cc:407] Connection header received
0927 18:46:10.862657 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:10.862659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:10.862691 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:10.862750 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:10.863045 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:10.863618 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:10.864547 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:10.864877 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:10.864956 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:10.865692 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:10.865696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:10.865699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:10.865710 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:10.865735 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:10.865749 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:10.865833 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:10.865936 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:46:10.865963 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260927 18:46:10.866174 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf867ca5d47e4290a54762b39c8f862c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf867ca5d47e4290a54762b39c8f862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } } }
I20260927 18:46:10.866256 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:10.866395 7329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:10.869272 7329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:10.873967 7329 catalog_manager.cc:1347] Generated new cluster ID: a0da081eba27452eb02613a6551753fd
I20260927 18:46:10.874014 7329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:10.890024 7329 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:10.891040 7329 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:10.905629 7329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf867ca5d47e4290a54762b39c8f862c: Generated new TSK 0
I20260927 18:46:10.906246 7329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:10.913923 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7242
W20260927 18:46:10.921914 7227 connection.cc:535] client connection to 127.0.0.1:45355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45355): Connection reset by peer (error 104)
2026-09-27T18:46:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:10 dist-test-slave-6lr6 krb5kdc[7341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:10 dist-test-slave-6lr6 krb5kdc[7341](info): set up 2 sockets
Sep 27 18:46:10 dist-test-slave-6lr6 krb5kdc[7341](info): commencing operation
krb5kdc: starting...
W20260927 18:46:11.852937 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:11 dist-test-slave-6lr6 krb5kdc[7341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:11.959233 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:12.060413 7354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:12.060645 7354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:12.060693 7354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:12.064525 7354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:12.064574 7354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:12.064592 7354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:12.064606 7354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:12.064632 7354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:12.069576 7354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:12.070452 7354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:12.071331 7354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:12.079002 7362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:12.078997 7359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:12.079155 7360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:12.079226 7354 server_base.cc:1034] running on GCE node
I20260927 18:46:12.079732 7354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:12.080708 7354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:12.081851 7354 hybrid_clock.cc:648] HybridClock initialized: now 1790534772081837 us; error 27 us; skew 500 ppm
Sep 27 18:46:12 dist-test-slave-6lr6 krb5kdc[7341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:12.084285 7354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:12.085031 7354 webserver.cc:460] Webserver started at http://127.0.0.1:39597/ using document root <none> and password file <none>
I20260927 18:46:12.085541 7354 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:12.085597 7354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:12.085776 7354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:12.087589 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "72ac88bf408b4ec09134d765ef6e212a"
format_stamp: "Formatted at 2026-09-27 18:46:12 on dist-test-slave-6lr6"
server_key: "3eca100c100e36d695722a82a13a0bb3"
server_key_iv: "bfa91fe74f3066928a2fd04a6afdda45"
server_key_version: "encryptionkey@0"
I20260927 18:46:12.088088 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "72ac88bf408b4ec09134d765ef6e212a"
format_stamp: "Formatted at 2026-09-27 18:46:12 on dist-test-slave-6lr6"
server_key: "3eca100c100e36d695722a82a13a0bb3"
server_key_iv: "bfa91fe74f3066928a2fd04a6afdda45"
server_key_version: "encryptionkey@0"
I20260927 18:46:12.091401 7354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:12.093480 7369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:12.094549 7354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:12.094653 7354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "72ac88bf408b4ec09134d765ef6e212a"
format_stamp: "Formatted at 2026-09-27 18:46:12 on dist-test-slave-6lr6"
server_key: "3eca100c100e36d695722a82a13a0bb3"
server_key_iv: "bfa91fe74f3066928a2fd04a6afdda45"
server_key_version: "encryptionkey@0"
I20260927 18:46:12.094736 7354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:12.098834 7354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:12.101271 7354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:12.101414 7354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:12.107306 7354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38139
I20260927 18:46:12.107327 7421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38139 every 8 connection(s)
I20260927 18:46:12.108091 7354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:12.110572 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:12.113441 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7354
I20260927 18:46:12.113579 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:12.113875 19374 external_mini_cluster.cc:1428] Setting key 14e03a263a241cfcbf5800a88b102199
I20260927 18:46:12.115926 7422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a: Bootstrap starting.
I20260927 18:46:12.118182 7422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:12.118530 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:12.115081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44147 (local address 127.0.0.1:38139)
0927 18:46:12.115448 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:12.115461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:12.115493 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:46:12.116102 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:12.116119 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:12.116411 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:12.116664 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:12.117429 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:12.117433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:12.117445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:12.117454 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:12.117504 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:12.117523 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:12.117623 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:12.117721 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:46:12.117852 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260927 18:46:12.118886 7422 log.cc:824] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:12.120575 7422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a: No bootstrap required, opened a new log
I20260927 18:46:12.121025 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:12.124503 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:12.123200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58059 (local address 127.0.0.1:38139)
0927 18:46:12.123299 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:12.123303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:12.123420 (+ 117us) server_negotiation.cc:407] Connection header received
0927 18:46:12.123539 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:12.123541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:12.123566 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:12.123628 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:12.124205 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:12.124207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:12.124209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:12.124216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:12.124232 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:12.124239 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:12.124298 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:12.124387 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 18:46:12.124413 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:46:12.124688 7422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac88bf408b4ec09134d765ef6e212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260927 18:46:12.124939 7422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:12.124995 7422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ac88bf408b4ec09134d765ef6e212a, State: Initialized, Role: FOLLOWER
I20260927 18:46:12.125406 7422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac88bf408b4ec09134d765ef6e212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260927 18:46:12.125540 7422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:12.125584 7422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:12.125656 7422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:12.126370 7422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac88bf408b4ec09134d765ef6e212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260927 18:46:12.126633 7422 leader_election.cc:302] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72ac88bf408b4ec09134d765ef6e212a; no voters:
I20260927 18:46:12.126854 7422 leader_election.cc:288] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:12.126951 7432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:12.127162 7432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [term 1 LEADER]: Becoming Leader. State: Replica: 72ac88bf408b4ec09134d765ef6e212a, State: Running, Role: LEADER
I20260927 18:46:12.127439 7432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac88bf408b4ec09134d765ef6e212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } }
I20260927 18:46:12.127873 7422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:12.128850 7433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72ac88bf408b4ec09134d765ef6e212a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac88bf408b4ec09134d765ef6e212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } } }
I20260927 18:46:12.128957 7433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:12.129238 7434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72ac88bf408b4ec09134d765ef6e212a. Latest consensus state: current_term: 1 leader_uuid: "72ac88bf408b4ec09134d765ef6e212a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72ac88bf408b4ec09134d765ef6e212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38139 } } }
I20260927 18:46:12.129330 7441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:12.129320 7434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:12.132165 7441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:12.136934 7441 catalog_manager.cc:1347] Generated new cluster ID: c6e32246252048728b101294cd6f8054
I20260927 18:46:12.137037 7441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:12.151418 7441 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:12.151675 7441 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:12.169144 7441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72ac88bf408b4ec09134d765ef6e212a: Generated new TSK 0
I20260927 18:46:12.169834 7441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:12.179232 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7354
2026-09-27T18:46:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:12 dist-test-slave-6lr6 krb5kdc[7453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:12 dist-test-slave-6lr6 krb5kdc[7453](info): set up 2 sockets
Sep 27 18:46:12 dist-test-slave-6lr6 krb5kdc[7453](info): commencing operation
krb5kdc: starting...
W20260927 18:46:13.124918 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:13 dist-test-slave-6lr6 krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:13.231669 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:13.334496 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:13.334743 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:13.334786 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:13.338716 7466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:13.338764 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:13.338783 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:13.338814 7466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:13.338838 7466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:13.343828 7466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:13.344714 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:13.345664 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:13.353196 7471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:13.353160 7472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:13.353233 7466 server_base.cc:1034] running on GCE node
W20260927 18:46:13.353168 7474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:13.353886 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:13.354846 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:13.355996 7466 hybrid_clock.cc:648] HybridClock initialized: now 1790534773355983 us; error 24 us; skew 500 ppm
Sep 27 18:46:13 dist-test-slave-6lr6 krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:13.358386 7466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:13.359135 7466 webserver.cc:460] Webserver started at http://127.0.0.1:34527/ using document root <none> and password file <none>
I20260927 18:46:13.359602 7466 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:13.359664 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:13.359843 7466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:13.361692 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04"
format_stamp: "Formatted at 2026-09-27 18:46:13 on dist-test-slave-6lr6"
server_key: "5e1d913fb35428d0844e44d082af6ce0"
server_key_iv: "9da5afc252a4e438c4f0e6b5a1a768bb"
server_key_version: "encryptionkey@0"
I20260927 18:46:13.362188 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04"
format_stamp: "Formatted at 2026-09-27 18:46:13 on dist-test-slave-6lr6"
server_key: "5e1d913fb35428d0844e44d082af6ce0"
server_key_iv: "9da5afc252a4e438c4f0e6b5a1a768bb"
server_key_version: "encryptionkey@0"
I20260927 18:46:13.365456 7466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:13.367529 7481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:13.368597 7466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:13.368700 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04"
format_stamp: "Formatted at 2026-09-27 18:46:13 on dist-test-slave-6lr6"
server_key: "5e1d913fb35428d0844e44d082af6ce0"
server_key_iv: "9da5afc252a4e438c4f0e6b5a1a768bb"
server_key_version: "encryptionkey@0"
I20260927 18:46:13.368788 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:13.379359 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:13.381978 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:13.382128 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:13.388145 7466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35473
I20260927 18:46:13.388190 7533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35473 every 8 connection(s)
I20260927 18:46:13.388965 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:13.391474 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:13.395879 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7466
I20260927 18:46:13.395993 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:13.396226 19374 external_mini_cluster.cc:1428] Setting key 7437bb15997e02faae646efaa88546ca
I20260927 18:46:13.396544 7534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04: Bootstrap starting.
I20260927 18:46:13.398746 7534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:13.399564 7534 log.cc:824] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:13.401436 7534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04: No bootstrap required, opened a new log
Sep 27 18:46:13 dist-test-slave-6lr6 krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:46:13.405377 7534 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 18:46:13.405606 7534 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:13.405673 7534 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1bbb8f3e6d949e8b1dcc316da8dba04, State: Initialized, Role: FOLLOWER
I20260927 18:46:13.406109 7534 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 18:46:13.406250 7534 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:13.406314 7534 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:13.406409 7534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:13.407150 7534 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 18:46:13.407490 7534 leader_election.cc:302] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1bbb8f3e6d949e8b1dcc316da8dba04; no voters:
I20260927 18:46:13.407822 7534 leader_election.cc:288] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:13.407888 7539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:13.408123 7539 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [term 1 LEADER]: Becoming Leader. State: Replica: b1bbb8f3e6d949e8b1dcc316da8dba04, State: Running, Role: LEADER
I20260927 18:46:13.408414 7539 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 18:46:13.409181 7534 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:13.409519 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1bbb8f3e6d949e8b1dcc316da8dba04. Latest consensus state: current_term: 1 leader_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } } }
I20260927 18:46:13.409626 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:13.409963 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bbb8f3e6d949e8b1dcc316da8dba04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } } }
I20260927 18:46:13.410023 7543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:13.410050 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:13.412034 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:13.397474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57459 (local address 127.0.0.1:35473)
0927 18:46:13.397823 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:13.397837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:13.397869 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:46:13.398443 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:13.398459 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:13.398777 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:13.399053 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:13.399879 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:13.400940 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:13.402084 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:13.402682 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:13.402801 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:13.405486 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:13.405491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:13.405503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:13.405527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:13.407074 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:13.407517 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:13.407521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:13.407525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:13.407652 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:13.407930 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:13.407933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:13.407934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:13.408407 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:13.408592 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:13.408894 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 18:46:13.409199 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260927 18:46:13.412674 7543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:13.414973 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:13.417775 7543 catalog_manager.cc:1347] Generated new cluster ID: e1937f727e59456ea5719a9d40d24a06
I20260927 18:46:13.417829 7543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:13.425916 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:13.417193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42755 (local address 127.0.0.1:35473)
0927 18:46:13.417287 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:13.417292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:13.417585 (+ 293us) server_negotiation.cc:407] Connection header received
0927 18:46:13.417707 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:13.417710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:13.417762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:13.417841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:13.418710 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:13.419446 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:13.420573 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:13.420928 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:13.421006 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:13.422654 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:13.422658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:13.422660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:13.422684 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:13.424169 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:13.424627 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:13.424631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:13.424632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:13.424711 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:13.425004 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:13.425007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:13.425009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:13.425208 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:13.425327 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:13.425630 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:46:13.425752 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 18:46:13.426687 7543 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:13.427539 7543 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:13.435011 7543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1bbb8f3e6d949e8b1dcc316da8dba04: Generated new TSK 0
I20260927 18:46:13.435595 7543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:13.445371 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7466
2026-09-27T18:46:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:13 dist-test-slave-6lr6 krb5kdc[7565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:13 dist-test-slave-6lr6 krb5kdc[7565](info): set up 2 sockets
Sep 27 18:46:13 dist-test-slave-6lr6 krb5kdc[7565](info): commencing operation
krb5kdc: starting...
W20260927 18:46:14.389516 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:14 dist-test-slave-6lr6 krb5kdc[7565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:14.498378 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:14.598515 7578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:14.598719 7578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:14.598757 7578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:14.602525 7578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:14.602569 7578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:14.602587 7578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:14.602602 7578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:14.602622 7578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:14.607396 7578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:14.608153 7578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:14.608970 7578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:14.616122 7584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:14.616232 7578 server_base.cc:1034] running on GCE node
W20260927 18:46:14.616122 7586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:14.616243 7583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:14.616811 7578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:14.617762 7578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:14.618906 7578 hybrid_clock.cc:648] HybridClock initialized: now 1790534774618887 us; error 34 us; skew 500 ppm
Sep 27 18:46:14 dist-test-slave-6lr6 krb5kdc[7565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:14.624387 7578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:14.625274 7578 webserver.cc:460] Webserver started at http://127.0.0.1:38571/ using document root <none> and password file <none>
I20260927 18:46:14.625746 7578 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:14.625805 7578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:14.625988 7578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:14.627748 7578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "f642cb4b704c4317bcbf32db435cfaa6"
format_stamp: "Formatted at 2026-09-27 18:46:14 on dist-test-slave-6lr6"
server_key: "d66d4caf4955f2cf26b24b7bbe7f5aef"
server_key_iv: "0ad19e6e24bf74a1283038b15680efbf"
server_key_version: "encryptionkey@0"
I20260927 18:46:14.628240 7578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "f642cb4b704c4317bcbf32db435cfaa6"
format_stamp: "Formatted at 2026-09-27 18:46:14 on dist-test-slave-6lr6"
server_key: "d66d4caf4955f2cf26b24b7bbe7f5aef"
server_key_iv: "0ad19e6e24bf74a1283038b15680efbf"
server_key_version: "encryptionkey@0"
I20260927 18:46:14.631477 7578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 18:46:14.633528 7593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:14.634584 7578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 18:46:14.634687 7578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "f642cb4b704c4317bcbf32db435cfaa6"
format_stamp: "Formatted at 2026-09-27 18:46:14 on dist-test-slave-6lr6"
server_key: "d66d4caf4955f2cf26b24b7bbe7f5aef"
server_key_iv: "0ad19e6e24bf74a1283038b15680efbf"
server_key_version: "encryptionkey@0"
I20260927 18:46:14.634778 7578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:14.644809 7578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:14.647078 7578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:14.647225 7578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:14.653266 7578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37401
I20260927 18:46:14.653283 7645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37401 every 8 connection(s)
I20260927 18:46:14.654057 7578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:14.656529 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:14.661428 7646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6: Bootstrap starting.
I20260927 18:46:14.662958 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7578
I20260927 18:46:14.663102 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:14.663419 19374 external_mini_cluster.cc:1428] Setting key fc476685637fd8e50c986151945570c5
I20260927 18:46:14.663825 7646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:14.664611 7646 log.cc:824] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:14.666378 7646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6: No bootstrap required, opened a new log
I20260927 18:46:14.670030 7646 raft_consensus.cc:348] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f642cb4b704c4317bcbf32db435cfaa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260927 18:46:14.670238 7646 raft_consensus.cc:374] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:14.670342 7646 raft_consensus.cc:729] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f642cb4b704c4317bcbf32db435cfaa6, State: Initialized, Role: FOLLOWER
I20260927 18:46:14.670787 7646 consensus_queue.cc:260] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f642cb4b704c4317bcbf32db435cfaa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260927 18:46:14.670912 7646 raft_consensus.cc:388] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:14.670981 7646 raft_consensus.cc:482] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:14.671074 7646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:14.671418 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:14.664383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35179 (local address 127.0.0.1:37401)
0927 18:46:14.664731 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:14.664749 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:14.664788 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:46:14.665500 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:14.665518 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:14.665814 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:14.666058 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:14.666352 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:14.667288 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:14.668462 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:14.668888 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:14.668997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:14.669747 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:14.669752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:14.669764 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:14.670101 (+ 337us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:14.670148 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:14.670168 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:14.670258 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:14.670399 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:46:14.670570 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260927 18:46:14.671849 7646 raft_consensus.cc:504] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f642cb4b704c4317bcbf32db435cfaa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260927 18:46:14.672134 7646 leader_election.cc:302] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f642cb4b704c4317bcbf32db435cfaa6; no voters:
I20260927 18:46:14.672421 7646 leader_election.cc:288] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:14.672483 7651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:14.672672 7651 raft_consensus.cc:686] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [term 1 LEADER]: Becoming Leader. State: Replica: f642cb4b704c4317bcbf32db435cfaa6, State: Running, Role: LEADER
I20260927 18:46:14.672988 7651 consensus_queue.cc:237] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f642cb4b704c4317bcbf32db435cfaa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260927 18:46:14.673527 7646 sys_catalog.cc:564] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:14.674072 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:14.674624 7652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f642cb4b704c4317bcbf32db435cfaa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f642cb4b704c4317bcbf32db435cfaa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } } }
I20260927 18:46:14.674615 7653 sys_catalog.cc:455] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f642cb4b704c4317bcbf32db435cfaa6. Latest consensus state: current_term: 1 leader_uuid: "f642cb4b704c4317bcbf32db435cfaa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f642cb4b704c4317bcbf32db435cfaa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } } }
I20260927 18:46:14.674736 7652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:14.674744 7653 sys_catalog.cc:458] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6 [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:14.678187 7668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:14.678249 7668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:14.678326 7660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:14.679067 7660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:14.681770 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:14.676939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56035 (local address 127.0.0.1:37401)
0927 18:46:14.677028 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:14.677033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:14.677290 (+ 257us) server_negotiation.cc:407] Connection header received
0927 18:46:14.677449 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:14.677452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:14.677491 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:14.677534 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:14.677892 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:14.678617 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:14.679653 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:14.679989 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:14.680097 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:14.681370 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:14.681374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:14.681377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:14.681390 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:14.681418 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:14.681431 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:14.681509 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:14.681603 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:46:14.681650 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:46:14.684082 7660 catalog_manager.cc:1347] Generated new cluster ID: fb6c8b6dfa6f4546bee69bc0efe5991c
I20260927 18:46:14.684129 7660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:14.694800 7660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:14.695595 7660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:14.707370 7660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f642cb4b704c4317bcbf32db435cfaa6: Generated new TSK 0
I20260927 18:46:14.707974 7660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:14.717072 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7578
2026-09-27T18:46:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:14 dist-test-slave-6lr6 krb5kdc[7677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:14 dist-test-slave-6lr6 krb5kdc[7677](info): set up 2 sockets
Sep 27 18:46:14 dist-test-slave-6lr6 krb5kdc[7677](info): commencing operation
krb5kdc: starting...
W20260927 18:46:15.660920 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:15 dist-test-slave-6lr6 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:15.765332 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:15.865410 7690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:15.865619 7690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:15.865672 7690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:15.869705 7690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:15.869776 7690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:15.869807 7690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:15.869838 7690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:15.869872 7690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:15.874819 7690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:15.875654 7690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:15.876528 7690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:15.883906 7695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:15.883937 7696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:15.883903 7698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:15.883984 7690 server_base.cc:1034] running on GCE node
I20260927 18:46:15.884534 7690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:15.885526 7690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:15.886680 7690 hybrid_clock.cc:648] HybridClock initialized: now 1790534775886663 us; error 29 us; skew 500 ppm
Sep 27 18:46:15 dist-test-slave-6lr6 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:15.889045 7690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:15.889792 7690 webserver.cc:460] Webserver started at http://127.0.0.1:42317/ using document root <none> and password file <none>
I20260927 18:46:15.890242 7690 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:15.890312 7690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:15.890475 7690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:15.892194 7690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "afb01a90c9394a138358a3183d6181a4"
format_stamp: "Formatted at 2026-09-27 18:46:15 on dist-test-slave-6lr6"
server_key: "a7b98e5970f036c4ed179b0276c0e073"
server_key_iv: "936b6f54b2b37a2b02c6f0b391a43744"
server_key_version: "encryptionkey@0"
I20260927 18:46:15.892685 7690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "afb01a90c9394a138358a3183d6181a4"
format_stamp: "Formatted at 2026-09-27 18:46:15 on dist-test-slave-6lr6"
server_key: "a7b98e5970f036c4ed179b0276c0e073"
server_key_iv: "936b6f54b2b37a2b02c6f0b391a43744"
server_key_version: "encryptionkey@0"
I20260927 18:46:15.895828 7690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:15.897887 7705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:15.898910 7690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:15.899010 7690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "afb01a90c9394a138358a3183d6181a4"
format_stamp: "Formatted at 2026-09-27 18:46:15 on dist-test-slave-6lr6"
server_key: "a7b98e5970f036c4ed179b0276c0e073"
server_key_iv: "936b6f54b2b37a2b02c6f0b391a43744"
server_key_version: "encryptionkey@0"
I20260927 18:46:15.899091 7690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:15.903208 7690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:15.905948 7690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:15.906095 7690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:15.912426 7690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46497
I20260927 18:46:15.912451 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46497 every 8 connection(s)
I20260927 18:46:15.913306 7690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:15.915925 7758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:15.919467 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7690
I20260927 18:46:15.919605 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:15.919893 19374 external_mini_cluster.cc:1428] Setting key 8d93a4735ada1ceec73db1285ceaca59
I20260927 18:46:15.921447 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4: Bootstrap starting.
I20260927 18:46:15.923508 7758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:15.924222 7758 log.cc:824] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:15.924566 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:15.921179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42047 (local address 127.0.0.1:46497)
0927 18:46:15.921592 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:15.921606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:15.921640 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:46:15.922224 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:15.922239 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:15.922494 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:15.922789 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:15.923468 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:15.923473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:15.923484 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:15.923494 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:15.923541 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:15.923565 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:15.923656 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:15.923781 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 18:46:15.923924 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260927 18:46:15.925858 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4: No bootstrap required, opened a new log
I20260927 18:46:15.926982 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:15.929435 7758 raft_consensus.cc:348] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb01a90c9394a138358a3183d6181a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260927 18:46:15.929688 7758 raft_consensus.cc:374] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:15.929749 7758 raft_consensus.cc:729] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb01a90c9394a138358a3183d6181a4, State: Initialized, Role: FOLLOWER
I20260927 18:46:15.930148 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:15.928770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41039 (local address 127.0.0.1:46497)
0927 18:46:15.928861 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:15.928865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:15.929024 (+ 159us) server_negotiation.cc:407] Connection header received
0927 18:46:15.929144 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:15.929147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:15.929175 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:15.929235 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:15.929821 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:15.929824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:15.929827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:15.929838 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:15.929860 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:15.929871 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:15.929937 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:15.930019 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 18:46:15.930052 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:46:15.930151 7758 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb01a90c9394a138358a3183d6181a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260927 18:46:15.930341 7758 raft_consensus.cc:388] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:15.930416 7758 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:15.930507 7758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:15.931327 7758 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb01a90c9394a138358a3183d6181a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260927 18:46:15.931617 7758 leader_election.cc:302] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afb01a90c9394a138358a3183d6181a4; no voters:
I20260927 18:46:15.931880 7758 leader_election.cc:288] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:15.931948 7768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:15.932147 7768 raft_consensus.cc:686] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [term 1 LEADER]: Becoming Leader. State: Replica: afb01a90c9394a138358a3183d6181a4, State: Running, Role: LEADER
I20260927 18:46:15.932454 7768 consensus_queue.cc:237] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb01a90c9394a138358a3183d6181a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260927 18:46:15.932891 7758 sys_catalog.cc:564] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:15.933851 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb01a90c9394a138358a3183d6181a4. Latest consensus state: current_term: 1 leader_uuid: "afb01a90c9394a138358a3183d6181a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb01a90c9394a138358a3183d6181a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } } }
I20260927 18:46:15.933961 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:15.934324 7777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:15.936851 7777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:15.934288 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afb01a90c9394a138358a3183d6181a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb01a90c9394a138358a3183d6181a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } } }
I20260927 18:46:15.937469 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:15.941592 7777 catalog_manager.cc:1347] Generated new cluster ID: 55795ed363f44f8bac87f9f473990a37
I20260927 18:46:15.941685 7777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:15.966192 7777 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:15.966462 7777 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:15.974017 7777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afb01a90c9394a138358a3183d6181a4: Generated new TSK 0
I20260927 18:46:15.974532 7777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:15.984055 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7690
2026-09-27T18:46:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1266 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:16 dist-test-slave-6lr6 krb5kdc[7789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:16 dist-test-slave-6lr6 krb5kdc[7789](info): set up 2 sockets
Sep 27 18:46:16 dist-test-slave-6lr6 krb5kdc[7789](info): commencing operation
krb5kdc: starting...
W20260927 18:46:16.924854 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:16 dist-test-slave-6lr6 krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:17.028517 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:17.130395 7802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:17.130615 7802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:17.130656 7802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:17.134528 7802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:17.134578 7802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:17.134598 7802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:17.134626 7802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:17.134652 7802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:17.139562 7802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:17.140455 7802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:17.141376 7802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:17.148782 7810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:17.148800 7808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:17.148805 7807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:17.148869 7802 server_base.cc:1034] running on GCE node
I20260927 18:46:17.149473 7802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:17.150434 7802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:17.151577 7802 hybrid_clock.cc:648] HybridClock initialized: now 1790534777151561 us; error 26 us; skew 500 ppm
Sep 27 18:46:17 dist-test-slave-6lr6 krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:17.154167 7802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:17.154934 7802 webserver.cc:460] Webserver started at http://127.0.0.1:40515/ using document root <none> and password file <none>
I20260927 18:46:17.155385 7802 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:17.155436 7802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:17.155587 7802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:17.157441 7802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "b8f220bb35494816a938a23d8616ab39"
format_stamp: "Formatted at 2026-09-27 18:46:17 on dist-test-slave-6lr6"
server_key: "44270fb6d7813fb91744a99dbed0c196"
server_key_iv: "347853baf61389d8a93fefc19c8d33e0"
server_key_version: "encryptionkey@0"
I20260927 18:46:17.157919 7802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "b8f220bb35494816a938a23d8616ab39"
format_stamp: "Formatted at 2026-09-27 18:46:17 on dist-test-slave-6lr6"
server_key: "44270fb6d7813fb91744a99dbed0c196"
server_key_iv: "347853baf61389d8a93fefc19c8d33e0"
server_key_version: "encryptionkey@0"
I20260927 18:46:17.161175 7802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:17.163240 7817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:17.164306 7802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:17.164410 7802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "b8f220bb35494816a938a23d8616ab39"
format_stamp: "Formatted at 2026-09-27 18:46:17 on dist-test-slave-6lr6"
server_key: "44270fb6d7813fb91744a99dbed0c196"
server_key_iv: "347853baf61389d8a93fefc19c8d33e0"
server_key_version: "encryptionkey@0"
I20260927 18:46:17.164494 7802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:17.191049 7802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:17.193858 7802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:17.194017 7802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:17.200275 7802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42221
I20260927 18:46:17.200287 7869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42221 every 8 connection(s)
I20260927 18:46:17.201155 7802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:17.202699 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7802
I20260927 18:46:17.202847 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:17.203137 19374 external_mini_cluster.cc:1428] Setting key 6e0d259cfdab15933d6e83b794faebbc
I20260927 18:46:17.204048 7870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:17.209967 7870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39: Bootstrap starting.
Sep 27 18:46:17 dist-test-slave-6lr6 krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:46:17.212266 7870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:17.213020 7870 log.cc:824] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:17.214744 7870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39: No bootstrap required, opened a new log
I20260927 18:46:17.216446 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:17.204378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:60559 (local address 127.0.0.1:42221)
0927 18:46:17.204758 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:17.204773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:17.204808 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:46:17.205451 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:17.205468 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:17.205796 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:17.206055 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:17.206905 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:17.207850 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:17.208981 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:17.209455 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:17.209562 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:17.212087 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:17.212092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:17.212104 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:17.212129 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:17.213822 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:17.214271 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:17.214276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:17.214282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:17.214387 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:17.214708 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:17.214711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:17.214713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:17.215085 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:17.215269 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:17.215578 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 18:46:17.215782 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260927 18:46:17.218225 7870 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f220bb35494816a938a23d8616ab39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260927 18:46:17.218452 7870 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:17.218509 7870 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f220bb35494816a938a23d8616ab39, State: Initialized, Role: FOLLOWER
I20260927 18:46:17.218897 7870 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f220bb35494816a938a23d8616ab39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260927 18:46:17.219046 7870 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:17.219110 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:17.219200 7870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:17.219244 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:17.220149 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f220bb35494816a938a23d8616ab39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260927 18:46:17.220505 7870 leader_election.cc:302] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f220bb35494816a938a23d8616ab39; no voters:
I20260927 18:46:17.220800 7870 leader_election.cc:288] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:17.220880 7879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:17.221058 7879 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [term 1 LEADER]: Becoming Leader. State: Replica: b8f220bb35494816a938a23d8616ab39, State: Running, Role: LEADER
I20260927 18:46:17.221436 7879 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f220bb35494816a938a23d8616ab39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260927 18:46:17.221928 7870 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:17.222422 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f220bb35494816a938a23d8616ab39. Latest consensus state: current_term: 1 leader_uuid: "b8f220bb35494816a938a23d8616ab39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f220bb35494816a938a23d8616ab39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } } }
I20260927 18:46:17.222594 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:17.222822 7880 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f220bb35494816a938a23d8616ab39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f220bb35494816a938a23d8616ab39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } } }
I20260927 18:46:17.222908 7880 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:17.223147 7889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:17.226125 7889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:17.231269 7889 catalog_manager.cc:1347] Generated new cluster ID: f8f79ca090094722b399a97d51c4f79f
I20260927 18:46:17.231355 7889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:17.232223 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:17.221313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:52753 (local address 127.0.0.1:42221)
0927 18:46:17.221408 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:17.221412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:17.223934 (+ 2522us) server_negotiation.cc:407] Connection header received
0927 18:46:17.224087 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:17.224090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:17.224133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:17.224210 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:17.224901 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:17.225624 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:17.226867 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:17.227192 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:17.227291 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:17.228788 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:17.228792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:17.228794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:17.228818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:17.230414 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:17.230845 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:17.230848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:17.230850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:17.230923 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:17.231304 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:17.231307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:17.231309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:17.231545 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:17.231679 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:17.231995 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:46:17.232081 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:46:17.255335 7889 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:17.256140 7889 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:17.263833 7889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8f220bb35494816a938a23d8616ab39: Generated new TSK 0
I20260927 18:46:17.264387 7889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:17.274096 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7802
W20260927 18:46:17.278793 7787 connection.cc:535] client connection to 127.0.0.1:42221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42221): Connection reset by peer (error 104)
2026-09-27T18:46:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:17 dist-test-slave-6lr6 krb5kdc[7901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:17 dist-test-slave-6lr6 krb5kdc[7901](info): set up 2 sockets
Sep 27 18:46:17 dist-test-slave-6lr6 krb5kdc[7901](info): commencing operation
krb5kdc: starting...
W20260927 18:46:18.220899 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:18 dist-test-slave-6lr6 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:18.327571 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:18.434299 7914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:18.434521 7914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:18.434557 7914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:18.438467 7914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:18.438513 7914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:18.438530 7914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:18.438544 7914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:18.438560 7914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:18.443548 7914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:18.444415 7914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:18.445351 7914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:18.452911 7922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:18.452886 7920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:18.453014 7914 server_base.cc:1034] running on GCE node
W20260927 18:46:18.452929 7919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:18.453588 7914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:18.454561 7914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:18.455717 7914 hybrid_clock.cc:648] HybridClock initialized: now 1790534778455704 us; error 29 us; skew 500 ppm
Sep 27 18:46:18 dist-test-slave-6lr6 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:18.458473 7914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:18.459236 7914 webserver.cc:460] Webserver started at http://127.0.0.1:38563/ using document root <none> and password file <none>
I20260927 18:46:18.459714 7914 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:18.459774 7914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:18.459941 7914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:18.461817 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "26d36d51121d4b8dbbfd984109c2b2ec"
format_stamp: "Formatted at 2026-09-27 18:46:18 on dist-test-slave-6lr6"
server_key: "0deb9fb9af9c4b1c80344623a4f6fa96"
server_key_iv: "395025a5908f8cfe8fade19fc54f983d"
server_key_version: "encryptionkey@0"
I20260927 18:46:18.462298 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "26d36d51121d4b8dbbfd984109c2b2ec"
format_stamp: "Formatted at 2026-09-27 18:46:18 on dist-test-slave-6lr6"
server_key: "0deb9fb9af9c4b1c80344623a4f6fa96"
server_key_iv: "395025a5908f8cfe8fade19fc54f983d"
server_key_version: "encryptionkey@0"
I20260927 18:46:18.465503 7914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:18.467603 7929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:18.468689 7914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:18.468797 7914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "26d36d51121d4b8dbbfd984109c2b2ec"
format_stamp: "Formatted at 2026-09-27 18:46:18 on dist-test-slave-6lr6"
server_key: "0deb9fb9af9c4b1c80344623a4f6fa96"
server_key_iv: "395025a5908f8cfe8fade19fc54f983d"
server_key_version: "encryptionkey@0"
I20260927 18:46:18.468878 7914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:18.488790 7914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:18.491372 7914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:18.491520 7914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:18.497711 7914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33271
I20260927 18:46:18.497723 7981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33271 every 8 connection(s)
I20260927 18:46:18.498512 7914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:18.500931 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:18.502266 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 7914
I20260927 18:46:18.502385 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:18.502638 19374 external_mini_cluster.cc:1428] Setting key 27c1b59385b66136aa1e6c098edcd0bc
I20260927 18:46:18.506788 7982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec: Bootstrap starting.
I20260927 18:46:18.509079 7982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:18.509815 7982 log.cc:824] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:18.511571 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:18.503808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:49497 (local address 127.0.0.1:33271)
0927 18:46:18.504189 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:18.504207 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:18.504245 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:46:18.505007 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:18.505028 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:18.505475 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:18.505813 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:18.506188 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:18.507555 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:18.508373 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:18.509009 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:18.509193 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:18.509887 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:18.509893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:18.509908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:18.510312 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:18.510359 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:18.510380 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:18.510500 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:18.510638 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:46:18.510762 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260927 18:46:18.511726 7982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec: No bootstrap required, opened a new log
I20260927 18:46:18.514153 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:18.515228 7982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260927 18:46:18.515439 7982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:18.515503 7982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d36d51121d4b8dbbfd984109c2b2ec, State: Initialized, Role: FOLLOWER
I20260927 18:46:18.515929 7982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260927 18:46:18.516088 7982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:18.516153 7982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:18.516245 7982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:18.517124 7982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260927 18:46:18.517438 7982 leader_election.cc:302] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d36d51121d4b8dbbfd984109c2b2ec; no voters:
I20260927 18:46:18.517736 7982 leader_election.cc:288] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:18.517812 7992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:18.518020 7992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [term 1 LEADER]: Becoming Leader. State: Replica: 26d36d51121d4b8dbbfd984109c2b2ec, State: Running, Role: LEADER
I20260927 18:46:18.518343 7992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } }
I20260927 18:46:18.518846 7982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:18.519829 7993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } } }
I20260927 18:46:18.519943 7993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:18.520299 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:18.515861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:42909 (local address 127.0.0.1:33271)
0927 18:46:18.515952 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:18.515955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:18.516118 (+ 163us) server_negotiation.cc:407] Connection header received
0927 18:46:18.516225 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:18.516228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:18.516271 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:18.516345 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:18.516713 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:18.517305 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:18.518379 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:18.518706 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:18.518813 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:18.519593 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:18.519597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:18.519599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:18.519610 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:18.519635 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:18.519648 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:18.519727 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:18.520145 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 18:46:18.520186 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:46:18.520334 8001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:18.520792 7994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d36d51121d4b8dbbfd984109c2b2ec. Latest consensus state: current_term: 1 leader_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d36d51121d4b8dbbfd984109c2b2ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33271 } } }
I20260927 18:46:18.520884 7994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:18.523059 8001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:18.528255 8001 catalog_manager.cc:1347] Generated new cluster ID: 048af40586844107bc1fead215153aae
I20260927 18:46:18.528312 8001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:18.546109 8001 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:18.547206 8001 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:18.558825 8001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d36d51121d4b8dbbfd984109c2b2ec: Generated new TSK 0
I20260927 18:46:18.559494 8001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:18.564488 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 7914
W20260927 18:46:18.568899 7899 connection.cc:535] client connection to 127.0.0.1:33271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33271): Connection reset by peer (error 104)
2026-09-27T18:46:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:18 dist-test-slave-6lr6 krb5kdc[8013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:18 dist-test-slave-6lr6 krb5kdc[8013](info): set up 2 sockets
Sep 27 18:46:18 dist-test-slave-6lr6 krb5kdc[8013](info): commencing operation
krb5kdc: starting...
W20260927 18:46:19.504909 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:19 dist-test-slave-6lr6 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:19.610967 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:19.714001 8026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:19.714241 8026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:19.714283 8026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:19.718199 8026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:19.718254 8026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:19.718276 8026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:19.718299 8026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:19.718322 8026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:19.723330 8026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:19.724174 8026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:19.725059 8026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:19.732640 8034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:19.732679 8031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:19.732689 8032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:19.732812 8026 server_base.cc:1034] running on GCE node
I20260927 18:46:19.733409 8026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:19.734346 8026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:19.735497 8026 hybrid_clock.cc:648] HybridClock initialized: now 1790534779735480 us; error 28 us; skew 500 ppm
Sep 27 18:46:19 dist-test-slave-6lr6 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:19.741293 8026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:19.742034 8026 webserver.cc:460] Webserver started at http://127.0.0.1:41047/ using document root <none> and password file <none>
I20260927 18:46:19.742496 8026 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:19.742548 8026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:19.742702 8026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:19.744500 8026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "bcd575acf2cb47dc99dda9716f02421b"
format_stamp: "Formatted at 2026-09-27 18:46:19 on dist-test-slave-6lr6"
server_key: "2b0e43a44440db5ba085ec2ab8bf692a"
server_key_iv: "b39c27bbb4b2288d8ee48f6d2abd0c78"
server_key_version: "encryptionkey@0"
I20260927 18:46:19.744976 8026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "bcd575acf2cb47dc99dda9716f02421b"
format_stamp: "Formatted at 2026-09-27 18:46:19 on dist-test-slave-6lr6"
server_key: "2b0e43a44440db5ba085ec2ab8bf692a"
server_key_iv: "b39c27bbb4b2288d8ee48f6d2abd0c78"
server_key_version: "encryptionkey@0"
I20260927 18:46:19.748255 8026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:19.750376 8041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:19.751461 8026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:19.751549 8026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "bcd575acf2cb47dc99dda9716f02421b"
format_stamp: "Formatted at 2026-09-27 18:46:19 on dist-test-slave-6lr6"
server_key: "2b0e43a44440db5ba085ec2ab8bf692a"
server_key_iv: "b39c27bbb4b2288d8ee48f6d2abd0c78"
server_key_version: "encryptionkey@0"
I20260927 18:46:19.751621 8026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:19.755632 8026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:19.758071 8026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:19.758208 8026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:19.764087 8026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38941
I20260927 18:46:19.764123 8093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38941 every 8 connection(s)
I20260927 18:46:19.764863 8026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:19.765244 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8026
I20260927 18:46:19.765399 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:19.765697 19374 external_mini_cluster.cc:1428] Setting key 0124698e6e6af1718aafc60092954300
I20260927 18:46:19.767697 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:19.770252 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:19.766922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:36449 (local address 127.0.0.1:38941)
0927 18:46:19.767274 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:19.767288 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:19.767319 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:46:19.767922 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:19.767939 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:19.768203 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:19.768452 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:19.769205 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:19.769209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:19.769220 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:19.769229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:19.769273 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:19.769290 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:19.769386 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:19.769484 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:46:19.769627 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260927 18:46:19.772615 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:19.773283 8094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b: Bootstrap starting.
I20260927 18:46:19.775751 8094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:19.776156 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:19.774687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:48561 (local address 127.0.0.1:38941)
0927 18:46:19.774791 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:19.774795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:19.775011 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:46:19.775106 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:19.775108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:19.775133 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:19.775192 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:19.775771 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:19.775774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:19.775775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:19.775784 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:19.775805 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:19.775815 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:19.775888 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:19.776020 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 18:46:19.776056 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:46:19.776546 8094 log.cc:824] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:19.778264 8094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b: No bootstrap required, opened a new log
I20260927 18:46:19.781378 8094 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd575acf2cb47dc99dda9716f02421b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38941 } }
I20260927 18:46:19.781566 8094 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:19.781605 8094 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd575acf2cb47dc99dda9716f02421b, State: Initialized, Role: FOLLOWER
I20260927 18:46:19.781927 8094 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd575acf2cb47dc99dda9716f02421b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38941 } }
I20260927 18:46:19.782027 8094 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:19.782068 8094 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:19.782130 8094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:19.783021 8094 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd575acf2cb47dc99dda9716f02421b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38941 } }
I20260927 18:46:19.783291 8094 leader_election.cc:302] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcd575acf2cb47dc99dda9716f02421b; no voters:
I20260927 18:46:19.783509 8094 leader_election.cc:288] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:19.783677 8104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:19.783938 8104 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [term 1 LEADER]: Becoming Leader. State: Replica: bcd575acf2cb47dc99dda9716f02421b, State: Running, Role: LEADER
I20260927 18:46:19.784266 8104 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd575acf2cb47dc99dda9716f02421b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38941 } }
I20260927 18:46:19.784538 8094 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:19.785669 8106 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcd575acf2cb47dc99dda9716f02421b. Latest consensus state: current_term: 1 leader_uuid: "bcd575acf2cb47dc99dda9716f02421b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd575acf2cb47dc99dda9716f02421b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38941 } } }
I20260927 18:46:19.785804 8106 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:19.786134 8105 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcd575acf2cb47dc99dda9716f02421b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcd575acf2cb47dc99dda9716f02421b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38941 } } }
I20260927 18:46:19.786233 8105 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:19.789203 8119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:19.789270 8119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:19.789333 8114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:19.789995 8114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:19.794519 8114 catalog_manager.cc:1347] Generated new cluster ID: 99db66e3f2f34569b4e2c003c2129e5c
I20260927 18:46:19.794577 8114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:19.804352 8114 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:19.804613 8114 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:19.811206 8114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcd575acf2cb47dc99dda9716f02421b: Generated new TSK 0
I20260927 18:46:19.811828 8114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:19.822525 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8026
2026-09-27T18:46:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:19 dist-test-slave-6lr6 krb5kdc[8125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:19 dist-test-slave-6lr6 krb5kdc[8125](info): set up 2 sockets
Sep 27 18:46:19 dist-test-slave-6lr6 krb5kdc[8125](info): commencing operation
krb5kdc: starting...
W20260927 18:46:20.765183 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:20 dist-test-slave-6lr6 krb5kdc[8125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:20.871074 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:20.985980 8138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:20.986202 8138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:20.986254 8138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:20.990111 8138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:20.990166 8138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:20.990206 8138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:20.990240 8138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:20.990273 8138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:20.995235 8138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:20.996104 8138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:20.997016 8138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:21.004550 8146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:21.004561 8143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:21.004559 8144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:21.004609 8138 server_base.cc:1034] running on GCE node
I20260927 18:46:21.005330 8138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:21.006274 8138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:21.007447 8138 hybrid_clock.cc:648] HybridClock initialized: now 1790534781007431 us; error 28 us; skew 500 ppm
Sep 27 18:46:21 dist-test-slave-6lr6 krb5kdc[8125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:21.009987 8138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:21.010792 8138 webserver.cc:460] Webserver started at http://127.0.0.1:33309/ using document root <none> and password file <none>
I20260927 18:46:21.011289 8138 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:21.011355 8138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:21.011548 8138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:21.013490 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "189ee159abae4f8199cb5fce721fc942"
format_stamp: "Formatted at 2026-09-27 18:46:21 on dist-test-slave-6lr6"
server_key: "b428ee0b9c39852b708712dff74a9bf1"
server_key_iv: "c584be38b0bd323cf06aba7bdc7842ab"
server_key_version: "encryptionkey@0"
I20260927 18:46:21.014017 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "189ee159abae4f8199cb5fce721fc942"
format_stamp: "Formatted at 2026-09-27 18:46:21 on dist-test-slave-6lr6"
server_key: "b428ee0b9c39852b708712dff74a9bf1"
server_key_iv: "c584be38b0bd323cf06aba7bdc7842ab"
server_key_version: "encryptionkey@0"
I20260927 18:46:21.017498 8138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:21.019630 8153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:21.020715 8138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:21.020824 8138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "189ee159abae4f8199cb5fce721fc942"
format_stamp: "Formatted at 2026-09-27 18:46:21 on dist-test-slave-6lr6"
server_key: "b428ee0b9c39852b708712dff74a9bf1"
server_key_iv: "c584be38b0bd323cf06aba7bdc7842ab"
server_key_version: "encryptionkey@0"
I20260927 18:46:21.020911 8138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:21.038192 8138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:21.040614 8138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:21.040776 8138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:21.047324 8138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39669
I20260927 18:46:21.047349 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39669 every 8 connection(s)
I20260927 18:46:21.048141 8138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:21.050591 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:21.055444 8206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942: Bootstrap starting.
I20260927 18:46:21.055476 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8138
I20260927 18:46:21.055588 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:21.055873 19374 external_mini_cluster.cc:1428] Setting key 9e02c421b613af015aad38f5dd60b1db
I20260927 18:46:21.057803 8206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:21.058444 8206 log.cc:824] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:21.060200 8206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942: No bootstrap required, opened a new log
Sep 27 18:46:21 dist-test-slave-6lr6 krb5kdc[8125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:46:21.064121 8206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189ee159abae4f8199cb5fce721fc942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39669 } }
I20260927 18:46:21.064347 8206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:21.064405 8206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189ee159abae4f8199cb5fce721fc942, State: Initialized, Role: FOLLOWER
I20260927 18:46:21.064828 8206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189ee159abae4f8199cb5fce721fc942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39669 } }
I20260927 18:46:21.064949 8206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:21.065004 8206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:21.065088 8206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:21.065861 8206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189ee159abae4f8199cb5fce721fc942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39669 } }
I20260927 18:46:21.066145 8206 leader_election.cc:302] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 189ee159abae4f8199cb5fce721fc942; no voters:
I20260927 18:46:21.066399 8206 leader_election.cc:288] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:21.066468 8211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:21.066629 8211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [term 1 LEADER]: Becoming Leader. State: Replica: 189ee159abae4f8199cb5fce721fc942, State: Running, Role: LEADER
I20260927 18:46:21.066962 8211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189ee159abae4f8199cb5fce721fc942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39669 } }
I20260927 18:46:21.067523 8206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:21.068475 8212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "189ee159abae4f8199cb5fce721fc942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189ee159abae4f8199cb5fce721fc942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39669 } } }
I20260927 18:46:21.068585 8212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:21.068857 8213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 189ee159abae4f8199cb5fce721fc942. Latest consensus state: current_term: 1 leader_uuid: "189ee159abae4f8199cb5fce721fc942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "189ee159abae4f8199cb5fce721fc942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39669 } } }
I20260927 18:46:21.068979 8213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:21.070541 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:21.057036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:48595 (local address 127.0.0.1:39669)
0927 18:46:21.057431 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:21.057446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:21.057480 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:46:21.058095 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:21.058111 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:21.058431 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:21.058706 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:21.059449 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:21.060413 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:21.061554 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:21.061998 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:21.062114 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:21.064810 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:21.064815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:21.064833 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:21.064860 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:21.066620 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:21.067261 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:21.067265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:21.067269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:21.067376 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:21.067727 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:21.067731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:21.067733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:21.068208 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:21.069253 (+ 1045us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:21.069430 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 18:46:21.069648 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":104,"threads_started":1}
W20260927 18:46:21.072306 8226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:21.072387 8226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:21.072449 8220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:21.073345 8220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:21.073601 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:21.079187 8220 catalog_manager.cc:1347] Generated new cluster ID: 818b300a040547feb357a661cd3aa72a
I20260927 18:46:21.079248 8220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:21.084090 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:21.075810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:57015 (local address 127.0.0.1:39669)
0927 18:46:21.075910 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:21.075915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:21.076248 (+ 333us) server_negotiation.cc:407] Connection header received
0927 18:46:21.076358 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:21.076362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:21.076419 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:21.076489 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:21.077193 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:21.077795 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:21.078583 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:21.078839 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:21.078939 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:21.080718 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:21.080721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:21.080724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:21.080748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:21.082326 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:21.082849 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:21.082852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:21.082854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:21.082936 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:21.083244 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:21.083247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:21.083249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:21.083449 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:21.083599 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:21.083834 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 18:46:21.083927 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:46:21.096237 8220 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:21.097198 8220 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:21.111348 8220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 189ee159abae4f8199cb5fce721fc942: Generated new TSK 0
I20260927 18:46:21.111985 8220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:21.119477 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8138
W20260927 18:46:21.123924 8123 connection.cc:535] client connection to 127.0.0.1:39669 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39669): Connection reset by peer (error 104)
2026-09-27T18:46:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:21 dist-test-slave-6lr6 krb5kdc[8237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:21 dist-test-slave-6lr6 krb5kdc[8237](info): set up 2 sockets
Sep 27 18:46:21 dist-test-slave-6lr6 krb5kdc[8237](info): commencing operation
krb5kdc: starting...
W20260927 18:46:22.064971 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:22 dist-test-slave-6lr6 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:22.169607 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:22.271519 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:22.271744 8250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:22.271806 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:22.275681 8250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:22.275734 8250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:22.275759 8250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:22.275781 8250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:22.275817 8250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:22.280823 8250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:22.281702 8250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:22.282579 8250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:22.290105 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:22.290117 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:22.290150 8250 server_base.cc:1034] running on GCE node
W20260927 18:46:22.290117 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:22.290824 8250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:22.291780 8250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:22.292932 8250 hybrid_clock.cc:648] HybridClock initialized: now 1790534782292920 us; error 27 us; skew 500 ppm
Sep 27 18:46:22 dist-test-slave-6lr6 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:22.295338 8250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:22.296114 8250 webserver.cc:460] Webserver started at http://127.0.0.1:37623/ using document root <none> and password file <none>
I20260927 18:46:22.296587 8250 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:22.296640 8250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:22.296788 8250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:22.298609 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "2d79d1bfaade48f986cb40e28411b6c3"
format_stamp: "Formatted at 2026-09-27 18:46:22 on dist-test-slave-6lr6"
server_key: "6550334eefc04c9655eab96c2af5fcf0"
server_key_iv: "c54a26d0c2a7b3b8ff4c294ba6dc5ed6"
server_key_version: "encryptionkey@0"
I20260927 18:46:22.299077 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "2d79d1bfaade48f986cb40e28411b6c3"
format_stamp: "Formatted at 2026-09-27 18:46:22 on dist-test-slave-6lr6"
server_key: "6550334eefc04c9655eab96c2af5fcf0"
server_key_iv: "c54a26d0c2a7b3b8ff4c294ba6dc5ed6"
server_key_version: "encryptionkey@0"
I20260927 18:46:22.302346 8250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:46:22.304409 8265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:22.305577 8250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:22.305675 8250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "2d79d1bfaade48f986cb40e28411b6c3"
format_stamp: "Formatted at 2026-09-27 18:46:22 on dist-test-slave-6lr6"
server_key: "6550334eefc04c9655eab96c2af5fcf0"
server_key_iv: "c54a26d0c2a7b3b8ff4c294ba6dc5ed6"
server_key_version: "encryptionkey@0"
I20260927 18:46:22.305748 8250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:22.315675 8250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:22.318238 8250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:22.318410 8250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:22.324597 8250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40985
I20260927 18:46:22.324607 8317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40985 every 8 connection(s)
I20260927 18:46:22.325457 8250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:22.327883 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:22.332759 8318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3: Bootstrap starting.
I20260927 18:46:22.333962 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8250
I20260927 18:46:22.334079 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:22.334321 19374 external_mini_cluster.cc:1428] Setting key 4f7a1964c5ea66bc7fc0934600dfd6da
I20260927 18:46:22.335134 8318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:22.335906 8318 log.cc:824] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:22.337908 8318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3: No bootstrap required, opened a new log
I20260927 18:46:22.341712 8318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79d1bfaade48f986cb40e28411b6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40985 } }
I20260927 18:46:22.341951 8318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:22.342010 8318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d79d1bfaade48f986cb40e28411b6c3, State: Initialized, Role: FOLLOWER
I20260927 18:46:22.342445 8318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79d1bfaade48f986cb40e28411b6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40985 } }
I20260927 18:46:22.342592 8318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:22.342667 8318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:22.342779 8318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:22.343277 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:22.335227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:58211 (local address 127.0.0.1:40985)
0927 18:46:22.335633 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:22.335650 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:22.335691 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:46:22.336442 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:22.336459 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:22.336838 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:22.337205 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:22.337605 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:22.339007 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:22.339826 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:22.340550 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:22.340700 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:22.341396 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:22.341402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:22.341416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:22.341848 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:22.341913 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:22.341943 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:22.342073 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:22.342241 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 18:46:22.342355 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20260927 18:46:22.343698 8318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79d1bfaade48f986cb40e28411b6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40985 } }
I20260927 18:46:22.343991 8318 leader_election.cc:302] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d79d1bfaade48f986cb40e28411b6c3; no voters:
I20260927 18:46:22.344240 8318 leader_election.cc:288] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:22.344326 8323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:22.344511 8323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [term 1 LEADER]: Becoming Leader. State: Replica: 2d79d1bfaade48f986cb40e28411b6c3, State: Running, Role: LEADER
I20260927 18:46:22.344828 8323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79d1bfaade48f986cb40e28411b6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40985 } }
I20260927 18:46:22.345366 8318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:22.345916 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:22.346027 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d79d1bfaade48f986cb40e28411b6c3. Latest consensus state: current_term: 1 leader_uuid: "2d79d1bfaade48f986cb40e28411b6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79d1bfaade48f986cb40e28411b6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40985 } } }
I20260927 18:46:22.346138 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:22.346444 8324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d79d1bfaade48f986cb40e28411b6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79d1bfaade48f986cb40e28411b6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40985 } } }
I20260927 18:46:22.346534 8324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:22.346778 8332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:22.349737 8332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:22.352728 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:22.347984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:43677 (local address 127.0.0.1:40985)
0927 18:46:22.348089 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:22.348093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:22.348524 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:46:22.348684 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:22.348687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:22.348729 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:22.348815 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:22.349056 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:22.349753 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:22.350932 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:22.351248 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:22.351350 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:22.352141 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:22.352145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:22.352148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:22.352160 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:22.352197 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:22.352211 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:22.352288 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:22.352580 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 18:46:22.352610 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 18:46:22.355091 8332 catalog_manager.cc:1347] Generated new cluster ID: 9eb9a3de3ed641e68b1b36adcdd19f0c
I20260927 18:46:22.355199 8332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:22.371002 8332 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:22.371924 8332 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:22.383459 8332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d79d1bfaade48f986cb40e28411b6c3: Generated new TSK 0
I20260927 18:46:22.384192 8332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:22.390111 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8250
W20260927 18:46:22.396847 8235 connection.cc:535] client connection to 127.0.0.1:40985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40985): Connection reset by peer (error 104)
2026-09-27T18:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:22 dist-test-slave-6lr6 krb5kdc[8349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:22 dist-test-slave-6lr6 krb5kdc[8349](info): set up 2 sockets
Sep 27 18:46:22 dist-test-slave-6lr6 krb5kdc[8349](info): commencing operation
krb5kdc: starting...
W20260927 18:46:23.336866 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:23 dist-test-slave-6lr6 krb5kdc[8349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:46:23.441689 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:23.544651 8362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:23.544870 8362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:23.544911 8362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:23.548715 8362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:23.548768 8362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:23.548789 8362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:23.548806 8362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:23.548823 8362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:23.553732 8362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:23.554508 8362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:23.555343 8362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:23.562863 8368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:23.562852 8367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:23.562855 8370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:23.563201 8362 server_base.cc:1034] running on GCE node
I20260927 18:46:23.563668 8362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:23.564600 8362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:23.565769 8362 hybrid_clock.cc:648] HybridClock initialized: now 1790534783565759 us; error 27 us; skew 500 ppm
Sep 27 18:46:23 dist-test-slave-6lr6 krb5kdc[8349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:23.568226 8362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:46:23.569012 8362 webserver.cc:460] Webserver started at http://127.0.0.1:44133/ using document root <none> and password file <none>
I20260927 18:46:23.569501 8362 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:23.569563 8362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:23.569727 8362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:23.571519 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "ad32cdebc24040a0a0bfcea091e2ecdc"
format_stamp: "Formatted at 2026-09-27 18:46:23 on dist-test-slave-6lr6"
server_key: "1d443c7c2891f5495322ff2e802bc739"
server_key_iv: "9c25f5e05cba78607ba81407cca7a8e0"
server_key_version: "encryptionkey@0"
I20260927 18:46:23.572009 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "ad32cdebc24040a0a0bfcea091e2ecdc"
format_stamp: "Formatted at 2026-09-27 18:46:23 on dist-test-slave-6lr6"
server_key: "1d443c7c2891f5495322ff2e802bc739"
server_key_iv: "9c25f5e05cba78607ba81407cca7a8e0"
server_key_version: "encryptionkey@0"
I20260927 18:46:23.575498 8362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:46:23.577723 8377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:23.578786 8362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:23.578886 8362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "ad32cdebc24040a0a0bfcea091e2ecdc"
format_stamp: "Formatted at 2026-09-27 18:46:23 on dist-test-slave-6lr6"
server_key: "1d443c7c2891f5495322ff2e802bc739"
server_key_iv: "9c25f5e05cba78607ba81407cca7a8e0"
server_key_version: "encryptionkey@0"
I20260927 18:46:23.578984 8362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:23.583266 8362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:23.585956 8362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:23.586118 8362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:23.592643 8362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44341
I20260927 18:46:23.592653 8429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44341 every 8 connection(s)
I20260927 18:46:23.593554 8362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:23.596079 8430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:23.596159 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8362
I20260927 18:46:23.596298 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:23.596575 19374 external_mini_cluster.cc:1428] Setting key 376e165602bbdf637908d504aa01ed13
I20260927 18:46:23.601740 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:23.597903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:44519 (local address 127.0.0.1:44341)
0927 18:46:23.598306 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:23.598324 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:23.598365 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:46:23.599077 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:23.599099 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:23.599452 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:23.599804 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:23.600439 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:23.600444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:23.600459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:23.600470 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:23.600525 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:23.600551 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:23.600677 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:23.600742 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 18:46:23.600887 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260927 18:46:23.602849 8430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc: Bootstrap starting.
I20260927 18:46:23.604326 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:23.605047 8430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:23.605782 8430 log.cc:824] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:23.607638 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:23.606196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.5:33049 (local address 127.0.0.1:44341)
0927 18:46:23.606313 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:23.606318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:23.606489 (+ 171us) server_negotiation.cc:407] Connection header received
0927 18:46:23.606595 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:23.606598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:23.606631 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:23.606721 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:23.607296 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:23.607299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:23.607302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:23.607315 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:23.607343 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:23.607358 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:23.607459 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:23.607492 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 18:46:23.607528 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:23.607752 8430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc: No bootstrap required, opened a new log
I20260927 18:46:23.611111 8430 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:46:23.611323 8430 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:23.611383 8430 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad32cdebc24040a0a0bfcea091e2ecdc, State: Initialized, Role: FOLLOWER
I20260927 18:46:23.611764 8430 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:46:23.611889 8430 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:23.611956 8430 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:23.612056 8430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:23.612893 8430 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:46:23.613233 8430 leader_election.cc:302] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad32cdebc24040a0a0bfcea091e2ecdc; no voters:
I20260927 18:46:23.613518 8430 leader_election.cc:288] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:23.613590 8440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:23.613793 8440 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [term 1 LEADER]: Becoming Leader. State: Replica: ad32cdebc24040a0a0bfcea091e2ecdc, State: Running, Role: LEADER
I20260927 18:46:23.614099 8440 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:46:23.614588 8430 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:23.615532 8441 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260927 18:46:23.615662 8441 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:23.616027 8449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:23.615988 8442 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad32cdebc24040a0a0bfcea091e2ecdc. Latest consensus state: current_term: 1 leader_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad32cdebc24040a0a0bfcea091e2ecdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260927 18:46:23.616339 8442 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:23.618847 8449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:23.623332 8449 catalog_manager.cc:1347] Generated new cluster ID: 77acdaea4ede4ced8f2ac6773a7df215
I20260927 18:46:23.623389 8449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:23.641250 8449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:23.641507 8449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:23.645941 8449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad32cdebc24040a0a0bfcea091e2ecdc: Generated new TSK 0
I20260927 18:46:23.646603 8449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:23.650175 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8362
2026-09-27T18:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:23 dist-test-slave-6lr6 krb5kdc[8461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:23 dist-test-slave-6lr6 krb5kdc[8461](info): set up 2 sockets
Sep 27 18:46:23 dist-test-slave-6lr6 krb5kdc[8461](info): commencing operation
krb5kdc: starting...
W20260927 18:46:24.592969 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:24 dist-test-slave-6lr6 krb5kdc[8461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:24.697947 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38517
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:38869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:38517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:24.797982 8474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:24.798188 8474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:24.798225 8474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:24.802044 8474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:24.802085 8474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:24.802102 8474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:24.802116 8474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:24.802129 8474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:24.807129 8474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:38869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:38517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:38517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:24.807916 8474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:24.808738 8474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:24.816042 8479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:24.816049 8482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:24.816047 8480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:24.816246 8474 server_base.cc:1034] running on GCE node
I20260927 18:46:24.816790 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:24.817754 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:24.818905 8474 hybrid_clock.cc:648] HybridClock initialized: now 1790534784818884 us; error 33 us; skew 500 ppm
Sep 27 18:46:24 dist-test-slave-6lr6 krb5kdc[8461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534784, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:24.825328 8474 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:24.826112 8474 webserver.cc:460] Webserver started at http://127.18.235.190:33699/ using document root <none> and password file <none>
I20260927 18:46:24.826582 8474 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:24.826643 8474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:24.826833 8474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:24.828634 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "26de3f5e30604d9586192737f70e684b"
format_stamp: "Formatted at 2026-09-27 18:46:24 on dist-test-slave-6lr6"
server_key: "762f955ab80931911b30f4c4bf720bda"
server_key_iv: "b46a1cfdeed9d474c70af75e373028f0"
server_key_version: "encryptionkey@0"
I20260927 18:46:24.829164 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "26de3f5e30604d9586192737f70e684b"
format_stamp: "Formatted at 2026-09-27 18:46:24 on dist-test-slave-6lr6"
server_key: "762f955ab80931911b30f4c4bf720bda"
server_key_iv: "b46a1cfdeed9d474c70af75e373028f0"
server_key_version: "encryptionkey@0"
I20260927 18:46:24.832468 8474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:46:24.834604 8489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:24.835676 8474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:24.835772 8474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "26de3f5e30604d9586192737f70e684b"
format_stamp: "Formatted at 2026-09-27 18:46:24 on dist-test-slave-6lr6"
server_key: "762f955ab80931911b30f4c4bf720bda"
server_key_iv: "b46a1cfdeed9d474c70af75e373028f0"
server_key_version: "encryptionkey@0"
I20260927 18:46:24.835866 8474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:24.850665 8474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:24.853397 8474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:24.853546 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:24.859638 8474 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:38517
I20260927 18:46:24.859649 8541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:38517 every 8 connection(s)
I20260927 18:46:24.860442 8474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:24.862345 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8474
I20260927 18:46:24.862475 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:24.862751 19374 external_mini_cluster.cc:1428] Setting key 5c05bf7092231bbb311adeee955821f0
I20260927 18:46:24.863160 8542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:24.868566 8542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b: Bootstrap starting.
Sep 27 18:46:24 dist-test-slave-6lr6 krb5kdc[8461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:24.870790 8542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:24.871517 8542 log.cc:824] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:24.873436 8542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b: No bootstrap required, opened a new log
I20260927 18:46:24.875300 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:24.863967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55165 (local address 127.18.235.190:38517)
0927 18:46:24.864372 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:24.864385 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:24.864418 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:46:24.865004 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:24.865019 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:24.865369 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:24.865643 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:24.866380 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:24.867287 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:24.868144 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:24.868588 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:24.868723 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:24.871195 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:24.871199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:24.871211 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:24.871235 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:24.872688 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:24.873279 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:24.873284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:24.873288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:24.873397 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:24.873713 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:24.873715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:24.873716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:24.874043 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:24.874192 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:24.874472 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 18:46:24.874667 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":135,"threads_started":1}
I20260927 18:46:24.877058 8542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26de3f5e30604d9586192737f70e684b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38517 } }
I20260927 18:46:24.877326 8542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:24.877380 8542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26de3f5e30604d9586192737f70e684b, State: Initialized, Role: FOLLOWER
I20260927 18:46:24.877789 8542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26de3f5e30604d9586192737f70e684b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38517 } }
I20260927 18:46:24.877951 8542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:24.877961 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:24.878015 8542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:24.878106 8542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:24.878855 8542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26de3f5e30604d9586192737f70e684b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38517 } }
I20260927 18:46:24.879165 8542 leader_election.cc:302] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26de3f5e30604d9586192737f70e684b; no voters:
I20260927 18:46:24.879431 8542 leader_election.cc:288] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:24.879518 8551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:24.879680 8551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [term 1 LEADER]: Becoming Leader. State: Replica: 26de3f5e30604d9586192737f70e684b, State: Running, Role: LEADER
I20260927 18:46:24.879956 8551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26de3f5e30604d9586192737f70e684b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38517 } }
I20260927 18:46:24.880539 8542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:24.881026 8553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26de3f5e30604d9586192737f70e684b. Latest consensus state: current_term: 1 leader_uuid: "26de3f5e30604d9586192737f70e684b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26de3f5e30604d9586192737f70e684b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38517 } } }
I20260927 18:46:24.881163 8553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:24.881459 8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26de3f5e30604d9586192737f70e684b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26de3f5e30604d9586192737f70e684b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 38517 } } }
I20260927 18:46:24.881567 8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:24.881826 8558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:24.884678 8558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:24.887966 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:24.879987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48403 (local address 127.18.235.190:38517)
0927 18:46:24.880070 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:24.880074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:24.880230 (+ 156us) server_negotiation.cc:407] Connection header received
0927 18:46:24.880320 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:24.880323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:24.880362 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:24.880421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:24.881214 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:24.881931 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:24.882958 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:24.883258 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:24.883356 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:24.884888 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:24.884892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:24.884895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:24.884917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:24.886236 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:24.886710 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:24.886713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:24.886715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:24.886801 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:24.887139 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:24.887143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:24.887144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:24.887337 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:24.887475 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:24.887733 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:46:24.887824 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 18:46:24.890389 8558 catalog_manager.cc:1347] Generated new cluster ID: bed21ba8aa8a44db87ff5987763b2559
I20260927 18:46:24.890437 8558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:24.904203 8558 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:24.905303 8558 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:24.910229 8558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26de3f5e30604d9586192737f70e684b: Generated new TSK 0
I20260927 18:46:24.910751 8558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:24.914523 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8474
W20260927 18:46:24.918607 8459 connection.cc:535] client connection to 127.18.235.190:38517 recv error: Network error: failed to read from TLS socket (remote: 127.18.235.190:38517): Connection reset by peer (error 104)
2026-09-27T18:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:24 dist-test-slave-6lr6 krb5kdc[8573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:24 dist-test-slave-6lr6 krb5kdc[8573](info): set up 2 sockets
Sep 27 18:46:24 dist-test-slave-6lr6 krb5kdc[8573](info): commencing operation
krb5kdc: starting...
W20260927 18:46:25.856958 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:25 dist-test-slave-6lr6 krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:25.961882 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:40881
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:40881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:26.062902 8586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:26.063124 8586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:26.063177 8586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:26.067032 8586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:26.067083 8586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:26.067108 8586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:26.067130 8586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:26.067158 8586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:26.072062 8586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:40881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:40881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:26.072944 8586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:26.073879 8586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:26.081537 8591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:26.081578 8594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:26.081537 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:26.081786 8586 server_base.cc:1034] running on GCE node
I20260927 18:46:26.082298 8586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:26.083280 8586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:26.084450 8586 hybrid_clock.cc:648] HybridClock initialized: now 1790534786084437 us; error 27 us; skew 500 ppm
Sep 27 18:46:26 dist-test-slave-6lr6 krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534786, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:26.087185 8586 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:26.087961 8586 webserver.cc:460] Webserver started at http://127.18.235.190:35741/ using document root <none> and password file <none>
I20260927 18:46:26.088444 8586 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:26.088505 8586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:26.088683 8586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:26.090546 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "d9615c74f7184e09aa06be73cd612d40"
format_stamp: "Formatted at 2026-09-27 18:46:26 on dist-test-slave-6lr6"
server_key: "51da2230fcbb4aac6bbbe41911d2a831"
server_key_iv: "ac22eb30858c7e974b6e38d6e63b97f8"
server_key_version: "encryptionkey@0"
I20260927 18:46:26.091042 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "d9615c74f7184e09aa06be73cd612d40"
format_stamp: "Formatted at 2026-09-27 18:46:26 on dist-test-slave-6lr6"
server_key: "51da2230fcbb4aac6bbbe41911d2a831"
server_key_iv: "ac22eb30858c7e974b6e38d6e63b97f8"
server_key_version: "encryptionkey@0"
I20260927 18:46:26.094360 8586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:26.096457 8601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:26.097615 8586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:26.097726 8586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "d9615c74f7184e09aa06be73cd612d40"
format_stamp: "Formatted at 2026-09-27 18:46:26 on dist-test-slave-6lr6"
server_key: "51da2230fcbb4aac6bbbe41911d2a831"
server_key_iv: "ac22eb30858c7e974b6e38d6e63b97f8"
server_key_version: "encryptionkey@0"
I20260927 18:46:26.097810 8586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:26.115520 8586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:26.118271 8586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:26.118425 8586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:26.124336 8586 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:40881
I20260927 18:46:26.124389 8653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:40881 every 8 connection(s)
I20260927 18:46:26.125202 8586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:26.126312 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8586
I20260927 18:46:26.126443 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:26.126731 19374 external_mini_cluster.cc:1428] Setting key 7bf0081ad69160864191ce333bf8821b
I20260927 18:46:26.128120 8654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:26.133689 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40: Bootstrap starting.
I20260927 18:46:26.134799 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:26.127933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49515 (local address 127.18.235.190:40881)
0927 18:46:26.128303 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:26.128317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:26.128351 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:46:26.128937 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:26.128952 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:26.129308 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:26.129568 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:26.129909 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:26.130861 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:26.131993 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:26.132433 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:26.132538 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:26.133341 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:26.133345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:26.133355 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:26.133686 (+ 331us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:26.133741 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:26.133763 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:26.133878 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:26.134030 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:46:26.134148 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260927 18:46:26.135902 8654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:26.136631 8654 log.cc:824] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:26.137425 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:26.138638 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40: No bootstrap required, opened a new log
I20260927 18:46:26.142215 8654 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9615c74f7184e09aa06be73cd612d40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40881 } }
I20260927 18:46:26.142472 8654 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:26.142549 8654 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9615c74f7184e09aa06be73cd612d40, State: Initialized, Role: FOLLOWER
I20260927 18:46:26.142943 8654 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9615c74f7184e09aa06be73cd612d40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40881 } }
I20260927 18:46:26.143074 8654 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:26.143138 8654 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:26.143234 8654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:26.143569 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:26.139459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44455 (local address 127.18.235.190:40881)
0927 18:46:26.139556 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:26.139560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:26.139696 (+ 136us) server_negotiation.cc:407] Connection header received
0927 18:46:26.139802 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:26.139805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:26.139862 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:26.139931 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:26.140113 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:26.141275 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:26.142021 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:26.142340 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:26.142441 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:26.143092 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:26.143096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:26.143098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:26.143110 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:26.143136 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:26.143149 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:26.143236 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:26.143425 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:46:26.143454 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:46:26.144171 8654 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9615c74f7184e09aa06be73cd612d40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40881 } }
I20260927 18:46:26.144491 8654 leader_election.cc:302] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9615c74f7184e09aa06be73cd612d40; no voters:
I20260927 18:46:26.144724 8654 leader_election.cc:288] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:26.144827 8664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:26.145001 8664 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [term 1 LEADER]: Becoming Leader. State: Replica: d9615c74f7184e09aa06be73cd612d40, State: Running, Role: LEADER
I20260927 18:46:26.145324 8664 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9615c74f7184e09aa06be73cd612d40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40881 } }
I20260927 18:46:26.145751 8654 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:26.146860 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9615c74f7184e09aa06be73cd612d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9615c74f7184e09aa06be73cd612d40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40881 } } }
I20260927 18:46:26.146888 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9615c74f7184e09aa06be73cd612d40. Latest consensus state: current_term: 1 leader_uuid: "d9615c74f7184e09aa06be73cd612d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9615c74f7184e09aa06be73cd612d40" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40881 } } }
I20260927 18:46:26.147025 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:26.147039 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:26.147424 8676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:26.149896 8676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:26.154470 8676 catalog_manager.cc:1347] Generated new cluster ID: df540af190ff49019c4bf6b096552786
I20260927 18:46:26.154529 8676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:26.172408 8676 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:26.173543 8676 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:26.186076 8676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9615c74f7184e09aa06be73cd612d40: Generated new TSK 0
I20260927 18:46:26.186796 8676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:26.191764 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8586
W20260927 18:46:26.196276 8571 connection.cc:535] client connection to 127.18.235.190:40881 recv error: Network error: failed to read from TLS socket (remote: 127.18.235.190:40881): Connection reset by peer (error 104)
2026-09-27T18:46:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:26 dist-test-slave-6lr6 krb5kdc[8685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:26 dist-test-slave-6lr6 krb5kdc[8685](info): set up 2 sockets
Sep 27 18:46:26 dist-test-slave-6lr6 krb5kdc[8685](info): commencing operation
krb5kdc: starting...
W20260927 18:46:27.141199 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:27 dist-test-slave-6lr6 krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:27.251121 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35225
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:27.355930 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:27.356166 8698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:27.356230 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:27.360122 8698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:27.360179 8698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:27.360211 8698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:27.360239 8698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:27.360270 8698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:27.365209 8698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:27.366081 8698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:27.366962 8698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:27.374469 8703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:27.374466 8706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:27.374485 8704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:27.374644 8698 server_base.cc:1034] running on GCE node
I20260927 18:46:27.375216 8698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:27.376166 8698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:27.377316 8698 hybrid_clock.cc:648] HybridClock initialized: now 1790534787377301 us; error 27 us; skew 500 ppm
Sep 27 18:46:27 dist-test-slave-6lr6 krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534787, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:27.379832 8698 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:27.380607 8698 webserver.cc:460] Webserver started at http://127.18.235.190:36861/ using document root <none> and password file <none>
I20260927 18:46:27.381093 8698 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:27.381196 8698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:27.381386 8698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:27.383201 8698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "06c3be2df48244fcb99108c4c7351c2b"
format_stamp: "Formatted at 2026-09-27 18:46:27 on dist-test-slave-6lr6"
server_key: "3ab2370dac2c67ae9c32088afca7ee1a"
server_key_iv: "52382adf322ba9495a12b2fd84fa7bdf"
server_key_version: "encryptionkey@0"
I20260927 18:46:27.383709 8698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "06c3be2df48244fcb99108c4c7351c2b"
format_stamp: "Formatted at 2026-09-27 18:46:27 on dist-test-slave-6lr6"
server_key: "3ab2370dac2c67ae9c32088afca7ee1a"
server_key_iv: "52382adf322ba9495a12b2fd84fa7bdf"
server_key_version: "encryptionkey@0"
I20260927 18:46:27.387032 8698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:46:27.389183 8713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:27.390256 8698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:46:27.390362 8698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "06c3be2df48244fcb99108c4c7351c2b"
format_stamp: "Formatted at 2026-09-27 18:46:27 on dist-test-slave-6lr6"
server_key: "3ab2370dac2c67ae9c32088afca7ee1a"
server_key_iv: "52382adf322ba9495a12b2fd84fa7bdf"
server_key_version: "encryptionkey@0"
I20260927 18:46:27.390445 8698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:27.394588 8698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:27.397379 8698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:27.397537 8698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:27.403687 8698 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:35225
I20260927 18:46:27.403697 8765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:35225 every 8 connection(s)
I20260927 18:46:27.404481 8698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:27.405402 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8698
I20260927 18:46:27.405532 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:27.405803 19374 external_mini_cluster.cc:1428] Setting key 10981d2786064d84b61822a0d68dc430
I20260927 18:46:27.407361 8766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:27.410353 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:27.407020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50743 (local address 127.18.235.190:35225)
0927 18:46:27.407432 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:27.407451 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:27.407490 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:46:27.408068 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:27.408083 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:27.408351 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:27.408637 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:27.409324 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:27.409328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:27.409339 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:27.409349 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:27.409396 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:27.409414 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:27.409530 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:27.409601 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:46:27.409738 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":133,"threads_started":1}
I20260927 18:46:27.412883 8766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b: Bootstrap starting.
I20260927 18:46:27.413038 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:27.415033 8766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:27.415745 8766 log.cc:824] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:27.416190 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:27.414821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34355 (local address 127.18.235.190:35225)
0927 18:46:27.414926 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:27.414930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:27.415083 (+ 153us) server_negotiation.cc:407] Connection header received
0927 18:46:27.415179 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:27.415182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:27.415211 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:27.415271 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:27.415849 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:27.415852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:27.415855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:27.415866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:27.415891 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:27.415902 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:27.415974 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:27.416057 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 18:46:27.416090 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:27.417498 8766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b: No bootstrap required, opened a new log
I20260927 18:46:27.420619 8766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3be2df48244fcb99108c4c7351c2b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35225 } }
I20260927 18:46:27.420811 8766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:27.420876 8766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c3be2df48244fcb99108c4c7351c2b, State: Initialized, Role: FOLLOWER
I20260927 18:46:27.421276 8766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3be2df48244fcb99108c4c7351c2b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35225 } }
I20260927 18:46:27.421409 8766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:27.421470 8766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:27.421562 8766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:27.422300 8766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3be2df48244fcb99108c4c7351c2b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35225 } }
I20260927 18:46:27.422595 8766 leader_election.cc:302] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c3be2df48244fcb99108c4c7351c2b; no voters:
I20260927 18:46:27.422843 8766 leader_election.cc:288] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:27.422930 8776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:27.423122 8776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [term 1 LEADER]: Becoming Leader. State: Replica: 06c3be2df48244fcb99108c4c7351c2b, State: Running, Role: LEADER
I20260927 18:46:27.423432 8776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3be2df48244fcb99108c4c7351c2b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35225 } }
I20260927 18:46:27.423874 8766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:27.424852 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c3be2df48244fcb99108c4c7351c2b. Latest consensus state: current_term: 1 leader_uuid: "06c3be2df48244fcb99108c4c7351c2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3be2df48244fcb99108c4c7351c2b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35225 } } }
I20260927 18:46:27.424968 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:27.425315 8777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c3be2df48244fcb99108c4c7351c2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c3be2df48244fcb99108c4c7351c2b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35225 } } }
I20260927 18:46:27.425375 8785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:27.428295 8785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:27.428431 8777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:27.433364 8785 catalog_manager.cc:1347] Generated new cluster ID: 52de5b5c633a4ddeb55f91cd775d0281
I20260927 18:46:27.433419 8785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:27.438511 8785 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:27.438841 8785 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:27.451999 8785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c3be2df48244fcb99108c4c7351c2b: Generated new TSK 0
I20260927 18:46:27.452641 8785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:27.460383 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8698
2026-09-27T18:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:27 dist-test-slave-6lr6 krb5kdc[8797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:27 dist-test-slave-6lr6 krb5kdc[8797](info): set up 2 sockets
Sep 27 18:46:27 dist-test-slave-6lr6 krb5kdc[8797](info): commencing operation
krb5kdc: starting...
W20260927 18:46:28.413003 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:28 dist-test-slave-6lr6 krb5kdc[8797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:28.519937 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:40965
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:40965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:28.620849 8810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:28.621060 8810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:28.621098 8810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:28.624888 8810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:28.624933 8810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:28.624950 8810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:28.624964 8810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:28.624977 8810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:28.629832 8810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:40965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:28.630606 8810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:28.631431 8810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:28.638913 8810 server_base.cc:1034] running on GCE node
W20260927 18:46:28.638934 8818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:28.639066 8816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:28.638944 8815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:28.639593 8810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:28.640555 8810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:28.641707 8810 hybrid_clock.cc:648] HybridClock initialized: now 1790534788641688 us; error 30 us; skew 500 ppm
Sep 27 18:46:28 dist-test-slave-6lr6 krb5kdc[8797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534788, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:28.644161 8810 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:28.644903 8810 webserver.cc:460] Webserver started at http://127.18.235.190:38557/ using document root <none> and password file <none>
I20260927 18:46:28.645394 8810 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:28.645452 8810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:28.645623 8810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:28.647401 8810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "0514575119804cebbf6c5d061dda5330"
format_stamp: "Formatted at 2026-09-27 18:46:28 on dist-test-slave-6lr6"
server_key: "8366959e61284eda618c5e734b6fbf23"
server_key_iv: "0f60af0346c00d3fc78f645133f380d5"
server_key_version: "encryptionkey@0"
I20260927 18:46:28.647893 8810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "0514575119804cebbf6c5d061dda5330"
format_stamp: "Formatted at 2026-09-27 18:46:28 on dist-test-slave-6lr6"
server_key: "8366959e61284eda618c5e734b6fbf23"
server_key_iv: "0f60af0346c00d3fc78f645133f380d5"
server_key_version: "encryptionkey@0"
I20260927 18:46:28.651204 8810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:28.653311 8825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:28.654392 8810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:28.654485 8810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "0514575119804cebbf6c5d061dda5330"
format_stamp: "Formatted at 2026-09-27 18:46:28 on dist-test-slave-6lr6"
server_key: "8366959e61284eda618c5e734b6fbf23"
server_key_iv: "0f60af0346c00d3fc78f645133f380d5"
server_key_version: "encryptionkey@0"
I20260927 18:46:28.654573 8810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:28.684993 8810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:28.687429 8810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:28.687584 8810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:28.693606 8810 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:40965
I20260927 18:46:28.693658 8877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:40965 every 8 connection(s)
I20260927 18:46:28.694465 8810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:28.696928 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:28.701848 8878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330: Bootstrap starting.
I20260927 18:46:28.703940 8878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:28.704561 8878 log.cc:824] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:28.704578 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8810
I20260927 18:46:28.704694 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:28.704936 19374 external_mini_cluster.cc:1428] Setting key a94cbfb44b0264f04ba6745961459509
I20260927 18:46:28.706420 8878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330: No bootstrap required, opened a new log
I20260927 18:46:28.710173 8878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0514575119804cebbf6c5d061dda5330" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40965 } }
I20260927 18:46:28.710430 8878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:28.710489 8878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0514575119804cebbf6c5d061dda5330, State: Initialized, Role: FOLLOWER
I20260927 18:46:28.710850 8878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0514575119804cebbf6c5d061dda5330" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40965 } }
I20260927 18:46:28.710981 8878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:28.711086 8878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:28.711189 8878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:28.712136 8878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0514575119804cebbf6c5d061dda5330" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40965 } }
I20260927 18:46:28.712489 8878 leader_election.cc:302] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0514575119804cebbf6c5d061dda5330; no voters:
I20260927 18:46:28.712785 8878 leader_election.cc:288] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:28.712870 8883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:28.713027 8883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [term 1 LEADER]: Becoming Leader. State: Replica: 0514575119804cebbf6c5d061dda5330, State: Running, Role: LEADER
Sep 27 18:46:28 dist-test-slave-6lr6 krb5kdc[8797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:28.713371 8883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0514575119804cebbf6c5d061dda5330" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40965 } }
I20260927 18:46:28.714012 8878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:28.714547 8885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0514575119804cebbf6c5d061dda5330. Latest consensus state: current_term: 1 leader_uuid: "0514575119804cebbf6c5d061dda5330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0514575119804cebbf6c5d061dda5330" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40965 } } }
I20260927 18:46:28.714720 8885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:28.715140 8890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:28.717041 8884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0514575119804cebbf6c5d061dda5330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0514575119804cebbf6c5d061dda5330" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40965 } } }
I20260927 18:46:28.717264 8884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:28.718551 8890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:28.719333 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:28.705875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58149 (local address 127.18.235.190:40965)
0927 18:46:28.706211 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:28.706225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:28.706270 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:46:28.706854 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:28.706871 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:28.707168 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:28.707429 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:28.708291 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:28.709225 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:28.710320 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:28.710841 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:28.710959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:28.713892 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:28.713899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:28.713915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:28.713948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:28.715964 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:28.716517 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:28.716523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:28.716529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:28.716643 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:28.716985 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:28.716989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:28.716991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:28.717509 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:28.717698 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:28.718001 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:46:28.718287 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260927 18:46:28.722028 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:28.724148 8890 catalog_manager.cc:1347] Generated new cluster ID: 23014f99366e4871810cdab1732d4335
I20260927 18:46:28.724217 8890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:28.732421 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:28.724310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39861 (local address 127.18.235.190:40965)
0927 18:46:28.724406 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:28.724410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:28.724459 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:46:28.724593 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:28.724596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:28.724642 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:28.724742 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:28.725505 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:28.726270 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:28.727371 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:28.727736 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:28.727838 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:28.729217 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:28.729222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:28.729224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:28.729248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:28.730758 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:28.731208 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:28.731211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:28.731213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:28.731293 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:28.731530 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:28.731533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:28.731534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:28.731720 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:28.731836 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:28.732037 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:46:28.732139 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:46:28.747278 8890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:28.748314 8890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:28.754863 8890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0514575119804cebbf6c5d061dda5330: Generated new TSK 0
I20260927 18:46:28.755429 8890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:28.759049 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8810
2026-09-27T18:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:28 dist-test-slave-6lr6 krb5kdc[8909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:28 dist-test-slave-6lr6 krb5kdc[8909](info): set up 2 sockets
Sep 27 18:46:28 dist-test-slave-6lr6 krb5kdc[8909](info): commencing operation
krb5kdc: starting...
W20260927 18:46:29.692937 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:29 dist-test-slave-6lr6 krb5kdc[8909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:29.796902 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:33191
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:32831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:33191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:29.898794 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:29.899016 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:29.899066 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:29.902913 8922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:29.902962 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:29.902987 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:29.903023 8922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:29.903055 8922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:29.907943 8922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:32831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:33191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:33191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:29.908767 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:29.909641 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:29.916967 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:29.916966 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:29.917042 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:29.917299 8922 server_base.cc:1034] running on GCE node
I20260927 18:46:29.917814 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:29.918762 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:29.919932 8922 hybrid_clock.cc:648] HybridClock initialized: now 1790534789919913 us; error 32 us; skew 500 ppm
Sep 27 18:46:29 dist-test-slave-6lr6 krb5kdc[8909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534789, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:29.925961 8922 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:29.926780 8922 webserver.cc:460] Webserver started at http://127.18.235.190:41581/ using document root <none> and password file <none>
I20260927 18:46:29.927256 8922 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:29.927325 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:29.927525 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:29.929504 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "58272e86de3a4a38ae42accc14ce962a"
format_stamp: "Formatted at 2026-09-27 18:46:29 on dist-test-slave-6lr6"
server_key: "cbd6b9d7f4ceee234cc6019bd387fc33"
server_key_iv: "dfc2030cb9823a4a1c32529cc5e52833"
server_key_version: "encryptionkey@0"
I20260927 18:46:29.930023 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "58272e86de3a4a38ae42accc14ce962a"
format_stamp: "Formatted at 2026-09-27 18:46:29 on dist-test-slave-6lr6"
server_key: "cbd6b9d7f4ceee234cc6019bd387fc33"
server_key_iv: "dfc2030cb9823a4a1c32529cc5e52833"
server_key_version: "encryptionkey@0"
I20260927 18:46:29.933323 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:29.935376 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:29.936415 8922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:29.936529 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "58272e86de3a4a38ae42accc14ce962a"
format_stamp: "Formatted at 2026-09-27 18:46:29 on dist-test-slave-6lr6"
server_key: "cbd6b9d7f4ceee234cc6019bd387fc33"
server_key_iv: "dfc2030cb9823a4a1c32529cc5e52833"
server_key_version: "encryptionkey@0"
I20260927 18:46:29.936614 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:29.987296 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:29.989671 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:29.989825 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:29.996083 8922 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:33191
I20260927 18:46:29.996099 8989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:33191 every 8 connection(s)
I20260927 18:46:29.996899 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:29.999358 8990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:30.001514 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 8922
I20260927 18:46:30.001639 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:30.001919 19374 external_mini_cluster.cc:1428] Setting key e1fc93fddee4c40966ec2bb1f9add619
I20260927 18:46:30.004810 8990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a: Bootstrap starting.
I20260927 18:46:30.006856 8990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:30.007670 8990 log.cc:824] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:30.009994 8990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a: No bootstrap required, opened a new log
I20260927 18:46:30.010628 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:30.003163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47177 (local address 127.18.235.190:33191)
0927 18:46:30.003570 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:30.003584 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:30.003617 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:46:30.004219 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:30.004234 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:30.004549 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:30.004862 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:30.005250 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:30.006158 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:30.007028 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:30.007723 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:30.007889 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:30.008802 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:30.008808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:30.008823 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:30.009286 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:30.009350 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:30.009380 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:30.009476 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:30.009561 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 18:46:30.009746 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260927 18:46:30.013412 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:30.013846 8990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58272e86de3a4a38ae42accc14ce962a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33191 } }
I20260927 18:46:30.014109 8990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:30.014163 8990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58272e86de3a4a38ae42accc14ce962a, State: Initialized, Role: FOLLOWER
I20260927 18:46:30.014613 8990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58272e86de3a4a38ae42accc14ce962a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33191 } }
I20260927 18:46:30.014775 8990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:30.014843 8990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:30.014940 8990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:30.015894 8990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58272e86de3a4a38ae42accc14ce962a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33191 } }
I20260927 18:46:30.016211 8990 leader_election.cc:302] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58272e86de3a4a38ae42accc14ce962a; no voters:
I20260927 18:46:30.016489 8990 leader_election.cc:288] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:30.016577 9000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:30.016728 9000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [term 1 LEADER]: Becoming Leader. State: Replica: 58272e86de3a4a38ae42accc14ce962a, State: Running, Role: LEADER
I20260927 18:46:30.017019 9000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58272e86de3a4a38ae42accc14ce962a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33191 } }
I20260927 18:46:30.017689 8990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:30.018554 9002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58272e86de3a4a38ae42accc14ce962a. Latest consensus state: current_term: 1 leader_uuid: "58272e86de3a4a38ae42accc14ce962a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58272e86de3a4a38ae42accc14ce962a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33191 } } }
I20260927 18:46:30.018676 9002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:30.019014 9009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:30.019001 9001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58272e86de3a4a38ae42accc14ce962a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58272e86de3a4a38ae42accc14ce962a" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 33191 } } }
I20260927 18:46:30.019095 9001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:30.019667 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:30.015519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34923 (local address 127.18.235.190:33191)
0927 18:46:30.015630 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:30.015634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:30.015783 (+ 149us) server_negotiation.cc:407] Connection header received
0927 18:46:30.015910 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:30.015911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:30.015945 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:30.016011 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:30.016259 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:30.016961 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:30.017833 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:30.018122 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:30.018255 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:30.019063 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:30.019067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:30.019069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:30.019080 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:30.019107 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:30.019119 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:30.019181 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:30.019490 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 18:46:30.019543 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:30.021872 9009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:30.026397 9009 catalog_manager.cc:1347] Generated new cluster ID: b2407b0fd5664677b9e828fa43bc65aa
I20260927 18:46:30.026455 9009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:30.033248 9009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:30.034027 9009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:30.048090 9009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58272e86de3a4a38ae42accc14ce962a: Generated new TSK 0
I20260927 18:46:30.048619 9009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:30.052327 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 8922
2026-09-27T18:46:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:30 dist-test-slave-6lr6 krb5kdc[9021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:30 dist-test-slave-6lr6 krb5kdc[9021](info): set up 2 sockets
Sep 27 18:46:30 dist-test-slave-6lr6 krb5kdc[9021](info): commencing operation
krb5kdc: starting...
W20260927 18:46:31.005021 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:31 dist-test-slave-6lr6 krb5kdc[9021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:31.113045 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39981
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:39981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:31.213967 9034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:31.214196 9034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:31.214248 9034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:31.218472 9034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:31.218521 9034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:31.218547 9034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:31.218569 9034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:31.218597 9034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:31.223554 9034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:39981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:39981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:31.224426 9034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:31.225342 9034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:31.232956 9039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:31.232990 9034 server_base.cc:1034] running on GCE node
W20260927 18:46:31.233002 9040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:31.232959 9042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:31.233718 9034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:31.234655 9034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:31.235808 9034 hybrid_clock.cc:648] HybridClock initialized: now 1790534791235792 us; error 27 us; skew 500 ppm
Sep 27 18:46:31 dist-test-slave-6lr6 krb5kdc[9021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534791, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:31.238309 9034 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:31.239089 9034 webserver.cc:460] Webserver started at http://127.18.235.190:33545/ using document root <none> and password file <none>
I20260927 18:46:31.239576 9034 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:31.239636 9034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:31.239800 9034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:31.241894 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d"
format_stamp: "Formatted at 2026-09-27 18:46:31 on dist-test-slave-6lr6"
server_key: "a00355255d3e5f79afb25eacd99c3e48"
server_key_iv: "1014282a506fae04b47e59dbdd57b969"
server_key_version: "encryptionkey@0"
I20260927 18:46:31.242398 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d"
format_stamp: "Formatted at 2026-09-27 18:46:31 on dist-test-slave-6lr6"
server_key: "a00355255d3e5f79afb25eacd99c3e48"
server_key_iv: "1014282a506fae04b47e59dbdd57b969"
server_key_version: "encryptionkey@0"
I20260927 18:46:31.245646 9034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:31.247751 9049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:31.248834 9034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:31.248939 9034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d"
format_stamp: "Formatted at 2026-09-27 18:46:31 on dist-test-slave-6lr6"
server_key: "a00355255d3e5f79afb25eacd99c3e48"
server_key_iv: "1014282a506fae04b47e59dbdd57b969"
server_key_version: "encryptionkey@0"
I20260927 18:46:31.249022 9034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:31.253211 9034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:31.255721 9034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:31.255870 9034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:31.262069 9034 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:39981
I20260927 18:46:31.262085 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:39981 every 8 connection(s)
I20260927 18:46:31.262826 9034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:31.265321 9102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:31.267402 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9034
I20260927 18:46:31.267522 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:31.267737 19374 external_mini_cluster.cc:1428] Setting key 8a297f0f7714755385987486f3b61462
I20260927 18:46:31.270838 9102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d: Bootstrap starting.
I20260927 18:46:31.272668 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:31.268939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39841 (local address 127.18.235.190:39981)
0927 18:46:31.269330 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:31.269347 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:31.269386 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:46:31.270169 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:31.270190 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:31.270511 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:31.270855 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:31.271427 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:31.271433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:31.271446 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:31.271456 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:31.271506 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:31.271528 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:31.271651 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:31.271714 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 18:46:31.271868 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260927 18:46:31.273260 9102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:31.273921 9102 log.cc:824] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:31.275079 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:31.275650 9102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d: No bootstrap required, opened a new log
I20260927 18:46:31.278203 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:31.276856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52177 (local address 127.18.235.190:39981)
0927 18:46:31.276963 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:31.276967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:31.277094 (+ 127us) server_negotiation.cc:407] Connection header received
0927 18:46:31.277216 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:31.277219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:31.277248 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:31.277308 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:31.277881 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:31.277884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:31.277887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:31.277898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:31.277922 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:31.277934 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:31.278000 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:31.278071 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:46:31.278104 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:46:31.279096 9102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39981 } }
I20260927 18:46:31.279302 9102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:31.279353 9102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06dfe2de7b8346edb9bdd6da8fd06a7d, State: Initialized, Role: FOLLOWER
I20260927 18:46:31.279703 9102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39981 } }
I20260927 18:46:31.279816 9102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:31.279876 9102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:31.279966 9102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:31.280725 9102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39981 } }
I20260927 18:46:31.281010 9102 leader_election.cc:302] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06dfe2de7b8346edb9bdd6da8fd06a7d; no voters:
I20260927 18:46:31.281330 9102 leader_election.cc:288] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:31.281394 9112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:31.281576 9112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [term 1 LEADER]: Becoming Leader. State: Replica: 06dfe2de7b8346edb9bdd6da8fd06a7d, State: Running, Role: LEADER
I20260927 18:46:31.281899 9112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39981 } }
I20260927 18:46:31.282415 9102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:31.283331 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39981 } } }
I20260927 18:46:31.283329 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06dfe2de7b8346edb9bdd6da8fd06a7d. Latest consensus state: current_term: 1 leader_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dfe2de7b8346edb9bdd6da8fd06a7d" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 39981 } } }
I20260927 18:46:31.283488 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:31.283543 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:31.283886 9121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:31.286638 9121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:31.291235 9121 catalog_manager.cc:1347] Generated new cluster ID: 97e1e109548c42bb8019ad9f8353a5d3
I20260927 18:46:31.291292 9121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:31.301389 9121 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:31.301656 9121 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:31.307138 9121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06dfe2de7b8346edb9bdd6da8fd06a7d: Generated new TSK 0
I20260927 18:46:31.307796 9121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:31.320009 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9034
2026-09-27T18:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1267 ms)
[----------] 18 tests from AuthTokenIssuingTest (23004 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:31 dist-test-slave-6lr6 krb5kdc[9133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:31 dist-test-slave-6lr6 krb5kdc[9133](info): set up 2 sockets
Sep 27 18:46:31 dist-test-slave-6lr6 krb5kdc[9133](info): commencing operation
krb5kdc: starting...
W20260927 18:46:32.268998 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:32.373740 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:40565
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:40565,127.18.235.189:42645,127.18.235.188:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:32.477277 9146 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:46:32.477507 9146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:32.477546 9146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:32.477598 9146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:32.481396 9146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:32.481442 9146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:32.481469 9146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:32.481491 9146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:32.481532 9146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:32.486649 9146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:40565,127.18.235.189:42645,127.18.235.188:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:40565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:32.487474 9146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:32.488307 9146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:32.496166 9154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:32.496153 9152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:32.496140 9151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:32.496227 9146 server_base.cc:1034] running on GCE node
I20260927 18:46:32.496874 9146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:32.497835 9146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:32.498996 9146 hybrid_clock.cc:648] HybridClock initialized: now 1790534792498986 us; error 26 us; skew 500 ppm
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:32.501387 9146 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:32.502101 9146 webserver.cc:460] Webserver started at http://127.18.235.190:34793/ using document root <none> and password file <none>
I20260927 18:46:32.502532 9146 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:32.502583 9146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:32.502727 9146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:32.504452 9146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "c4127511aa294e18af87a23c051483e2"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "d60e4cc6c78f222c357c3a9879400101"
server_key_iv: "a16fdf8ae0d0dbffbec7e86d77a367d0"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.504907 9146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "c4127511aa294e18af87a23c051483e2"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "d60e4cc6c78f222c357c3a9879400101"
server_key_iv: "a16fdf8ae0d0dbffbec7e86d77a367d0"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.508075 9146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:46:32.510164 9161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:32.511199 9146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:32.511292 9146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "c4127511aa294e18af87a23c051483e2"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "d60e4cc6c78f222c357c3a9879400101"
server_key_iv: "a16fdf8ae0d0dbffbec7e86d77a367d0"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.511385 9146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:32.522562 9146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:32.525171 9146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:32.525315 9146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:32.531211 9146 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:40565
I20260927 18:46:32.531432 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:40565 every 8 connection(s)
I20260927 18:46:32.532003 9146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:32.534363 9214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:32.537673 9214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:32.538498 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9146
I20260927 18:46:32.538630 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:32.538904 19374 external_mini_cluster.cc:1428] Setting key fc2466eceda508061f5610b2536a2b2b
WARNING: no policy specified for kudu/127.18.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.189@KRBTEST.COM" created.
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:32.552685 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.539674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:40565 (local address 127.0.0.1:42470)
0927 18:46:32.540103 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.540146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.540922 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.541528 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.541533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.541915 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.542119 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.542131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.543227 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.543230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.543909 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.543917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.544488 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.544492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.544607 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.545035 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.545049 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.548513 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.550219 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.550228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.550236 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.550515 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.550768 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.550771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.550773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.550905 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.551445 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.551451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.551596 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.551773 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.552120 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":85,"thread_start_us":123,"threads_started":1}
I20260927 18:46:32.552685 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.539675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.18.235.190:40565)
0927 18:46:32.540121 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.540136 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.540239 (+ 103us) server_negotiation.cc:407] Connection header received
0927 18:46:32.541200 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.541204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.541369 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.541517 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.542249 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.543118 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.544013 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.544358 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.544490 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.548656 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.548662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.548675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.548699 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.550110 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.550601 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.550605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.550606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.550695 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.550988 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.550990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.550991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.551335 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.551430 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.551814 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.552036 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":346,"spinlock_wait_cycles":13056,"thread_start_us":135,"threads_started":1}
I20260927 18:46:32.554829 9214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:32.555416 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.555207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57308)
0927 18:46:32.555310 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.555369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:46:32.555586 9163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
I20260927 18:46:32.556265 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.556078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57310)
0927 18:46:32.556156 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.556196 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 18:46:32.556420 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:42645: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab.
Entry for principal kudu/127.18.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab.
I20260927 18:46:32.561923 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:42645
--webserver_interface=127.18.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab
--principal=kudu/127.18.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:40565,127.18.235.189:42645,127.18.235.188:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:46:32.605654 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } attempt: 1
I20260927 18:46:32.606323 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.606095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57314)
0927 18:46:32.606210 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.606260 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 18:46:32.606933 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.606727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57318)
0927 18:46:32.606839 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.606881 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 18:46:32.607039 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:42645: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
W20260927 18:46:32.661854 9221 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:46:32.662078 9221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:32.662109 9221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:32.662166 9221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:32.665994 9221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:32.666036 9221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:32.666062 9221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:32.666091 9221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:32.666119 9221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:32.670914 9221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:40565,127.18.235.189:42645,127.18.235.188:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab
--principal=kudu/127.18.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:32.671728 9221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:32.672554 9221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:32.675177 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } attempt: 2
I20260927 18:46:32.675859 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.675566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57334)
0927 18:46:32.675689 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.675749 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:46:32.676667 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.676460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57340)
0927 18:46:32.676567 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.676613 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:46:32.676805 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:42645: Network error: Client connection negotiation failed: client connection to 127.18.235.189:42645: connect: Connection refused (error 111)
W20260927 18:46:32.679987 9229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:32.679993 9226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:32.679996 9227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:32.680061 9221 server_base.cc:1034] running on GCE node
I20260927 18:46:32.680604 9221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:32.681663 9221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:32.682825 9221 hybrid_clock.cc:648] HybridClock initialized: now 1790534792682806 us; error 28 us; skew 500 ppm
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:32.685364 9221 init.cc:377] Logged in from keytab as kudu/127.18.235.189@KRBTEST.COM (short username kudu)
I20260927 18:46:32.686178 9221 webserver.cc:460] Webserver started at http://127.18.235.189:37327/ using document root <none> and password file <none>
I20260927 18:46:32.686637 9221 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:32.686697 9221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:32.686857 9221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:32.688612 9221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data/instance:
uuid: "8b9f492c836047c5bf21855e905b5beb"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "19dc2511d06dfea4f68ff36d84db2be4"
server_key_iv: "adb82fe746d0ccbdeaaf4a392327ff1e"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.689097 9221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal/instance:
uuid: "8b9f492c836047c5bf21855e905b5beb"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "19dc2511d06dfea4f68ff36d84db2be4"
server_key_iv: "adb82fe746d0ccbdeaaf4a392327ff1e"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.692266 9221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:32.694331 9236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:32.695360 9221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:32.695461 9221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal
uuid: "8b9f492c836047c5bf21855e905b5beb"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "19dc2511d06dfea4f68ff36d84db2be4"
server_key_iv: "adb82fe746d0ccbdeaaf4a392327ff1e"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.695539 9221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:32.706753 9221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:32.709095 9221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:32.709267 9221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:32.715206 9221 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.189:42645
I20260927 18:46:32.715461 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.189:42645 every 8 connection(s)
I20260927 18:46:32.715991 9221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
I20260927 18:46:32.716248 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9221
I20260927 18:46:32.716351 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-1/wal/instance
I20260927 18:46:32.716650 19374 external_mini_cluster.cc:1428] Setting key 33f60f3bfa47d48edca5d947aef101ce
I20260927 18:46:32.718873 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:32.722205 9289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.188@KRBTEST.COM" created.
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:32.735669 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.723784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.18.235.190:40565)
0927 18:46:32.723930 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.723935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.724658 (+ 723us) server_negotiation.cc:407] Connection header received
0927 18:46:32.725615 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.725618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.725674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.725740 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.727712 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.728224 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.729821 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.730045 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.730122 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.732578 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.732580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.732582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.732606 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.733808 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.734382 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.734385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.734386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.734456 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.734742 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.734744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.734745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.734906 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.734977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.735466 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.735556 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:46:32.736464 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.724136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:40565 (local address 127.0.0.1:42474)
0927 18:46:32.724528 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.724566 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.725423 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.726800 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.726806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.727087 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.727578 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.727596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.728346 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.728349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.729692 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.729704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.730181 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.730185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.730387 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.730891 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.730911 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.732424 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.733951 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.733956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.733967 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.734295 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.734519 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.734521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.734522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.734674 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.734999 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.735004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.735271 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.735414 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.735644 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260927 18:46:32.737600 9289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab.
Entry for principal kudu/127.18.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab.
I20260927 18:46:32.739497 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:35989
--webserver_interface=127.18.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab
--principal=kudu/127.18.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:40565,127.18.235.189:42645,127.18.235.188:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:32.746142 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.737990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57356)
0927 18:46:32.738092 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.738105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.738174 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.738644 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.738647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.738804 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.738897 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.738904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.739875 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.739878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.740607 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.740617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.741345 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.741348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.741416 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.741785 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.741798 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.742994 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.744849 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.744852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.744853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.745016 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.745313 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.745315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.745316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.745385 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.745826 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.745828 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.745892 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.745962 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.746044 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 18:46:32.746551 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.738078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.18.235.189:42645)
0927 18:46:32.738349 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.738368 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.738384 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:46:32.738415 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.738418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.738543 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.738636 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.738994 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.739767 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.740706 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.741240 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.741344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.743121 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.743131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.743143 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.743168 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.744752 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.745136 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.745142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.745144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.745229 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.745464 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.745468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.745470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.745737 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.745867 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.746057 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.746191 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":124,"threads_started":1}
I20260927 18:46:32.747937 9289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:32.748518 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.748266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55524)
0927 18:46:32.748380 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.748456 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 18:46:32.748694 9239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
I20260927 18:46:32.749418 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.749212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55536)
0927 18:46:32.749308 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.749363 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 18:46:32.749557 9289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:35989: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
I20260927 18:46:32.767958 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } attempt: 3
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:32.775724 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.768371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.18.235.189:42645)
0927 18:46:32.768457 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.768461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.768526 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:46:32.768635 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.768638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.768674 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.768731 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.769494 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.769964 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.770678 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.771302 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.771393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.773239 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.773242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.773244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.773275 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.774402 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.774753 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.774756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.774758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.774827 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.775057 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.775059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.775061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.775203 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.775291 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.775514 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.775603 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:46:32.775725 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.768335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57370)
0927 18:46:32.768457 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.768471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.768559 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.768780 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.768782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.768923 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.769391 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.769399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.770056 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.770059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.770577 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.770584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.771405 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.771409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.771505 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.771877 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.771888 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.773095 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.774494 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.774497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.774499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.774672 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.774904 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.774907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.774909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.774982 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.775287 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.775290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.775376 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.775458 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.775574 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:46:32.776351 9214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:32.776958 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.776758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55552)
0927 18:46:32.776857 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.776904 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:46:32.777604 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.777394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55558)
0927 18:46:32.777517 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.777557 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 18:46:32.777731 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:35989: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
I20260927 18:46:32.798753 9289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } attempt: 1
I20260927 18:46:32.799377 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.799116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55570)
0927 18:46:32.799226 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.799300 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:46:32.800033 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.799807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55584)
0927 18:46:32.799916 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.799971 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 18:46:32.800165 9289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:35989: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
I20260927 18:46:32.808897 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } attempt: 1
I20260927 18:46:32.809512 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.809291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55588)
0927 18:46:32.809402 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.809451 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:46:32.810120 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.809921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55594)
0927 18:46:32.810025 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.810068 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:46:32.810258 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:35989: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
W20260927 18:46:32.845664 9295 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:46:32.845927 9295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:32.845956 9295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:32.846000 9295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:32.849818 9295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:32.849865 9295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:32.849885 9295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:32.849903 9295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:32.849923 9295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:32.854820 9295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:40565,127.18.235.189:42645,127.18.235.188:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab
--principal=kudu/127.18.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:32.855644 9295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:32.856513 9295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:32.864063 9304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:32.864051 9301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:32.864053 9302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:32.864264 9295 server_base.cc:1034] running on GCE node
I20260927 18:46:32.864825 9295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:32.865818 9295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:32.866987 9295 hybrid_clock.cc:648] HybridClock initialized: now 1790534792866974 us; error 32 us; skew 500 ppm
I20260927 18:46:32.868305 9289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } attempt: 2
I20260927 18:46:32.868961 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.868706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55610)
0927 18:46:32.868835 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.868891 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:32.869541 9295 init.cc:377] Logged in from keytab as kudu/127.18.235.188@KRBTEST.COM (short username kudu)
I20260927 18:46:32.869686 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.869462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55622)
0927 18:46:32.869569 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.869619 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 18:46:32.869850 9289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:35989: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
I20260927 18:46:32.870357 9295 webserver.cc:460] Webserver started at http://127.18.235.188:40333/ using document root <none> and password file <none>
I20260927 18:46:32.870832 9295 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:32.870888 9295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:32.871081 9295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:32.873000 9295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data/instance:
uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "28453cbf579add050ae17745f5967b83"
server_key_iv: "c75444c439006d0ac561f2c3d227e0aa"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.873588 9295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal/instance:
uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "28453cbf579add050ae17745f5967b83"
server_key_iv: "c75444c439006d0ac561f2c3d227e0aa"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.876843 9295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:32.879001 9311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:32.880046 9295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:32.880147 9295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal
uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad"
format_stamp: "Formatted at 2026-09-27 18:46:32 on dist-test-slave-6lr6"
server_key: "28453cbf579add050ae17745f5967b83"
server_key_iv: "c75444c439006d0ac561f2c3d227e0aa"
server_key_version: "encryptionkey@0"
I20260927 18:46:32.880228 9295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:32.885426 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } attempt: 2
I20260927 18:46:32.886023 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.885781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55628)
0927 18:46:32.885911 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.885960 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 18:46:32.886667 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.886479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55634)
0927 18:46:32.886574 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.886617 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:46:32.886806 9214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:35989: Network error: Client connection negotiation failed: client connection to 127.18.235.188:35989: connect: Connection refused (error 111)
I20260927 18:46:32.896675 9295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:32.899176 9295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:32.899338 9295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:32.905089 9295 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.188:35989
I20260927 18:46:32.905349 9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.188:35989 every 8 connection(s)
I20260927 18:46:32.905946 9295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
I20260927 18:46:32.908504 9364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:32.911723 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:32.914439 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9295
I20260927 18:46:32.914539 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534532155427-19374-0/minicluster-data/master-2/wal/instance
I20260927 18:46:32.914808 19374 external_mini_cluster.cc:1428] Setting key 026f16957db0f72f20cb5d6fdfbc51a9
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:46:32.923439 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.913180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.18.235.190:40565)
0927 18:46:32.913284 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.913287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.914046 (+ 759us) server_negotiation.cc:407] Connection header received
0927 18:46:32.914980 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.914983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.915035 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.915104 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.916258 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.916750 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.917777 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.917990 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.918076 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.920363 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.920367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.920370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.920396 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.921682 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.922156 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.922159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.922161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.922231 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.922519 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.922522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.922523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.922693 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.922773 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.923197 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.923302 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:46:32.924060 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.913550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:40565 (local address 127.0.0.1:42476)
0927 18:46:32.913925 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.913961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.914767 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.915260 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.915266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.915685 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.916130 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.916144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.916856 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.916860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.917671 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.917680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.918122 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.918126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.918270 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.918709 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.918724 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.920232 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.921774 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.921782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.921791 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.922071 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.922307 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.922309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.922311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.922437 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.922787 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.922793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.922995 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.923141 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.923408 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260927 18:46:32.924391 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.915376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.18.235.188:35989)
0927 18:46:32.915866 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.915874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.915891 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:46:32.915922 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.915925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.916061 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.916185 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.916877 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.917666 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.918455 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.918773 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.918858 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.921423 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.921428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.921436 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.921455 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.922734 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.923108 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.923112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.923114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.923181 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.923424 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.923427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.923429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.923737 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.923925 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.924108 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.924221 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":92,"threads_started":1}
I20260927 18:46:32.925819 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:32.933171 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.926191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:42645 (local address 127.0.0.1:57382)
0927 18:46:32.926319 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.926330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.926394 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.926657 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.926660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.926843 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.926923 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.926928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.927698 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.927701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.928254 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.928263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.928720 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.928724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.928835 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.929204 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.929217 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.930410 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.931872 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.931875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.931877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.932051 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.932288 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.932291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.932293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.932364 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.932727 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.932729 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.932811 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.932914 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.933026 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:46:32.933243 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.926339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.18.235.189:42645)
0927 18:46:32.926459 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.926463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.926477 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:46:32.926517 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.926520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.926560 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.926627 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.927031 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.927602 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.928356 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.928630 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.928702 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.930532 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.930535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.930537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.930561 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.931772 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.932135 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.932138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.932140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.932210 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.932461 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.932465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.932467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.932631 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.932725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.932964 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.933075 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:46:32.933845 9364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:32.933974 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.926221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.18.235.190:40565)
0927 18:46:32.926332 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.926335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.926526 (+ 191us) server_negotiation.cc:407] Connection header received
0927 18:46:32.926626 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.926629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.926667 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.926739 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.927301 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.927777 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.928503 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.928706 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.928796 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.931077 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.931080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.931082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.931102 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.932327 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.932680 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.932683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.932685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.932760 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.933000 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.933003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.933005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.933225 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.933430 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.933734 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.933843 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:32.940940 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.934211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55644)
0927 18:46:32.934296 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.934310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.934378 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.934583 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.934585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.934720 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.934800 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.934807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.935678 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.935681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.936159 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.936166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.936599 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.936602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.936700 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.937071 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.937084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.938314 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.939751 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.939753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.939754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.939920 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.940143 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.940146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.940148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.940214 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.940513 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.940515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.940588 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.940668 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.940781 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":17}
I20260927 18:46:32.940950 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.934318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.18.235.188:35989)
0927 18:46:32.934401 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.934405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.934416 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:46:32.934444 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.934446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.934475 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.934545 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.934893 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.935582 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.936261 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.936511 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.936592 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.938429 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.938432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.938434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.938453 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.939667 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.940004 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.940007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.940009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.940072 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.940284 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.940287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.940289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.940437 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.940512 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.940718 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.940832 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":23}
I20260927 18:46:32.942252 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.935004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.18.235.189:42645)
0927 18:46:32.935102 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.935106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.935118 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:46:32.935206 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.935208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.935239 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.935297 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.935904 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.936339 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.937020 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.937279 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.937355 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.939507 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.939510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.939513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.939536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.940737 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.941138 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.941141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.941143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.941226 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.941529 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.941532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.941533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.941732 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.941895 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.942058 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.942136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:46:32.944234 9364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: Bootstrap starting.
I20260927 18:46:32.945257 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:32.946756 9364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:32.946491 9188 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c4127511aa294e18af87a23c051483e2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42488
I20260927 18:46:32.947410 9364 log.cc:824] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:32.949280 9364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: No bootstrap required, opened a new log
I20260927 18:46:32.952940 9364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:32.953188 9364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e57e6f0acf4c62b8f4b101d8c1cfad, State: Initialized, Role: FOLLOWER
I20260927 18:46:32.953504 9364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:32.953913 9370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:32.954022 9370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:32.954596 9364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:46:32.959203 9381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:32.959316 9381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:32.957798 9295 master_runner.cc:186] Error getting master registration for 127.18.235.190:40565: OK, instance_id { permanent_uuid: "c4127511aa294e18af87a23c051483e2" instance_seqno: 1790534792529641 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:46:32.959977 9295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:46:32.960999 9289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } attempt: 3
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:46:32.968370 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.961369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55656)
0927 18:46:32.961468 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.961481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.961555 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.961769 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.961771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.961914 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.961981 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.961985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.962621 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.962624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.963114 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.963121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.963509 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.963512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.963588 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.963999 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.964011 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.965206 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.966958 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.966962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.966964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.967191 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.967467 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.967470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.967472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.967549 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.967937 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.967940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.968022 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.968110 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.968221 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:46:32.968926 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.961422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.18.235.188:35989)
0927 18:46:32.961507 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.961511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.961523 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:46:32.961648 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.961651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.961688 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.961745 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.962075 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.962531 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.963209 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.963420 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.963499 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.965367 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.965371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.965374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.965398 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.966856 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.967281 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.967294 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.967296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.967378 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.967623 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.967626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.967628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.967847 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.967939 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.968689 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.968798 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:46:32.971609 9289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: Bootstrap starting.
I20260927 18:46:32.973995 9289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:32.974749 9289 log.cc:824] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:32.976616 9289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: No bootstrap required, opened a new log
I20260927 18:46:32.980249 9289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:32.980435 9289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9f492c836047c5bf21855e905b5beb, State: Initialized, Role: FOLLOWER
I20260927 18:46:32.980732 9289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:32.981218 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:32.981438 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:32.981954 9289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:46:32.986800 9395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:32.986897 9395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:32.985001 9221 master_runner.cc:186] Error getting master registration for 127.18.235.190:40565: OK, instance_id { permanent_uuid: "c4127511aa294e18af87a23c051483e2" instance_seqno: 1790534792529641 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:46:32.988117 9221 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:46:32.988174 9214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } attempt: 3
Sep 27 18:46:32 dist-test-slave-6lr6 krb5kdc[9133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534792, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:46:32.997220 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.988550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:35989 (local address 127.0.0.1:55658)
0927 18:46:32.988706 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:32.988719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:32.988800 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:32.989025 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:32.989028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:32.989232 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:32.989332 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.989339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.990291 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.990294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.991107 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:32.991119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.991682 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.991685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:32.991787 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.992269 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:32.992284 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:32.993798 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:32.995654 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.995658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.995660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.995900 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.996196 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:32.996199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:32.996201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:32.996299 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:32.996708 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:32.996711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:32.996807 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 18:46:32.996904 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:46:32.997027 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 18:46:32.997416 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:32.988606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.18.235.188:35989)
0927 18:46:32.988698 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:32.988701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:32.988761 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:46:32.988879 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:32.988881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:32.988920 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:32.988998 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:32.989452 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.990183 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.991215 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:32.991562 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:32.991659 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:32.993930 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:32.993934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:32.993937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:32.993961 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:32.995535 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.996000 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.996003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.996005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.996093 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:32.996398 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:32.996402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:32.996404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:32.996616 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:32.996711 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:32.996982 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 18:46:32.997084 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:46:33.000376 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: Bootstrap starting.
I20260927 18:46:33.002820 9214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:33.003592 9214 log.cc:824] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:33.005630 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: No bootstrap required, opened a new log
I20260927 18:46:33.008448 9214 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:33.008682 9214 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4127511aa294e18af87a23c051483e2, State: Initialized, Role: FOLLOWER
I20260927 18:46:33.009006 9214 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:33.009568 9398 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.009779 9398 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:33.010316 9214 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:33.010380 9188 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:46:33.010474 9188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:33.011536 9188 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:33.013216 9263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4127511aa294e18af87a23c051483e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b9f492c836047c5bf21855e905b5beb"
I20260927 18:46:33.013448 9263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:33.012391 9188 leader_election.cc:288] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [CANDIDATE]: Term 1 election: Requested vote from peers 8b9f492c836047c5bf21855e905b5beb (127.18.235.189:42645), 39e57e6f0acf4c62b8f4b101d8c1cfad (127.18.235.188:35989)
I20260927 18:46:33.014659 9263 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4127511aa294e18af87a23c051483e2 in term 1.
I20260927 18:46:33.015028 9338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4127511aa294e18af87a23c051483e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad"
I20260927 18:46:33.015225 9338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:33.016369 9338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4127511aa294e18af87a23c051483e2 in term 1.
I20260927 18:46:33.017139 9164 leader_election.cc:302] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e57e6f0acf4c62b8f4b101d8c1cfad, c4127511aa294e18af87a23c051483e2; no voters:
I20260927 18:46:33.017602 9398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:33.017815 9398 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [term 1 LEADER]: Becoming Leader. State: Replica: c4127511aa294e18af87a23c051483e2, State: Running, Role: LEADER
I20260927 18:46:33.018121 9398 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } }
I20260927 18:46:33.021974 9146 master_runner.cc:263] No leader master found from master c4127511aa294e18af87a23c051483e2
I20260927 18:46:33.022029 9146 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:46:33.023180 9398 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4127511aa294e18af87a23c051483e2. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.023313 9398 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:33.026428 9409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:33.026558 9167 tablet.cc:2369] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:33.027036 9409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:33.027143 9419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:33.027918 9419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:33.049901 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:33.019116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.18.235.189:42645)
0927 18:46:33.019221 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.019226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.030045 (+ 10819us) server_negotiation.cc:407] Connection header received
0927 18:46:33.030482 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.030485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.030531 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.030577 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.034412 (+ 3835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:33.035137 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:33.036225 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:33.036584 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:33.036652 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:33.042768 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:33.042772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:33.042775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:33.042800 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:33.044368 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:33.047712 (+ 3344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:33.047715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:33.047717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:33.047799 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:33.048064 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:33.048067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:33.048068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:33.048283 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:33.048387 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:33.049636 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0927 18:46:33.049749 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:46:33.052678 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:33.019778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.18.235.190:40565)
0927 18:46:33.019872 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.019876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.030223 (+ 10347us) server_negotiation.cc:407] Connection header received
0927 18:46:33.030702 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.030705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.030743 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.030787 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.032430 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:33.033220 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:33.036785 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:33.037145 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:33.037213 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:33.045192 (+ 7979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:33.045195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:33.045198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:33.045222 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:33.046750 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:33.048455 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:33.048459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:33.048460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:33.048536 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:33.049000 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:33.049003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:33.049005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:33.049217 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:33.049316 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:33.052420 (+ 3104us) server_negotiation.cc:299] Negotiation successful
0927 18:46:33.052534 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:46:33.054111 9338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [term 1 FOLLOWER]: Refusing update from remote peer c4127511aa294e18af87a23c051483e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:33.054764 9398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:33.058646 9370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4127511aa294e18af87a23c051483e2. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.058763 9370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:33.058935 9263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [term 1 FOLLOWER]: Refusing update from remote peer c4127511aa294e18af87a23c051483e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:33.059463 9398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:33.062991 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4127511aa294e18af87a23c051483e2. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.063094 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:33.072486 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:33.021384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.18.235.188:35989)
0927 18:46:33.021477 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.021480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.042622 (+ 21142us) server_negotiation.cc:407] Connection header received
0927 18:46:33.044857 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.044861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.044909 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.044955 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.050218 (+ 5263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:33.050961 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:33.063394 (+ 12433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:33.063733 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:33.063806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:33.065482 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:33.065485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:33.065488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:33.065513 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:33.067023 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:33.069192 (+ 2169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:33.069196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:33.069198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:33.069291 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:33.070592 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:33.070595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:33.070597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:33.070795 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:33.070896 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:33.072229 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0927 18:46:33.072347 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:46:33.075474 9414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.075587 9414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:33.076656 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.076750 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:33.077126 9421 mvcc.cc:205] Tried to move back new op lower bound from 7334030512290615296 to 7334030512219398144. Current Snapshot: MvccSnapshot[applied={T|T < 7334030512290615296}]
I20260927 18:46:33.077966 9398 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.078058 9398 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:33.078408 9424 mvcc.cc:205] Tried to move back new op lower bound from 7334030512290615296 to 7334030512219398144. Current Snapshot: MvccSnapshot[applied={T|T < 7334030512290615296}]
I20260927 18:46:33.079113 9370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4127511aa294e18af87a23c051483e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4127511aa294e18af87a23c051483e2" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 40565 } } peers { permanent_uuid: "8b9f492c836047c5bf21855e905b5beb" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 42645 } } peers { permanent_uuid: "39e57e6f0acf4c62b8f4b101d8c1cfad" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 35989 } } }
I20260927 18:46:33.079224 9370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:33.079447 9422 mvcc.cc:205] Tried to move back new op lower bound from 7334030512290615296 to 7334030512219398144. Current Snapshot: MvccSnapshot[applied={T|T < 7334030512290615296}]
I20260927 18:46:33.081974 9419 catalog_manager.cc:1347] Generated new cluster ID: 6de3773553434891a05bafad8e6088eb
I20260927 18:46:33.082023 9419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:33.097052 9419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:33.098196 9419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:33.113094 9419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4127511aa294e18af87a23c051483e2: Generated new TSK 0
I20260927 18:46:33.113811 9419 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:46:33.145524 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.143718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.18.235.189:42645)
0927 18:46:33.143933 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.143937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.144438 (+ 501us) server_negotiation.cc:407] Connection header received
0927 18:46:33.144627 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.144631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.144677 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.144729 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.145399 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57416: BlockingRecv error: recv got EOF from 127.0.0.1:57416 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260927 18:46:33.145790 9217 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.144024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.18.235.190:40565)
0927 18:46:33.144140 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.144144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.144512 (+ 368us) server_negotiation.cc:407] Connection header received
0927 18:46:33.144653 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.144656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.144695 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.144740 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.145683 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42502: BlockingRecv error: recv got EOF from 127.0.0.1:42502 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 18:46:33.146656 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.143610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.18.235.188:35989)
0927 18:46:33.143844 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.143849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.145929 (+ 2080us) server_negotiation.cc:407] Connection header received
0927 18:46:33.145974 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.145977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.146019 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.146084 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.146549 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55674: BlockingRecv error: recv got EOF from 127.0.0.1:55674 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260927 18:46:33.149183 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.148216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.18.235.189:42645)
0927 18:46:33.148321 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.148325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.148468 (+ 143us) server_negotiation.cc:407] Connection header received
0927 18:46:33.148591 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.148594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.148652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.148713 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.149069 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57424: BlockingRecv error: recv got EOF from 127.0.0.1:57424 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 18:46:33.153017 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.152168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.18.235.189:42645)
0927 18:46:33.152249 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.152267 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.152392 (+ 125us) server_negotiation.cc:407] Connection header received
0927 18:46:33.152487 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.152490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.152529 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.152590 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.152934 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57430: BlockingRecv error: recv got EOF from 127.0.0.1:57430 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 18:46:33.157305 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.156371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.18.235.189:42645)
0927 18:46:33.156470 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.156474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.156660 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:46:33.156749 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.156773 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.156812 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.156871 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.157225 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57442: BlockingRecv error: recv got EOF from 127.0.0.1:57442 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 18:46:33.163574 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.162679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.18.235.189:42645)
0927 18:46:33.162806 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.162810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.162952 (+ 142us) server_negotiation.cc:407] Connection header received
0927 18:46:33.163056 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.163059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.163096 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.163155 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.163495 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57450: BlockingRecv error: recv got EOF from 127.0.0.1:57450 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:46:33.173879 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.172936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.18.235.189:42645)
0927 18:46:33.173019 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.173023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.173242 (+ 219us) server_negotiation.cc:407] Connection header received
0927 18:46:33.173354 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.173358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.173395 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.173454 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.173800 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57456: BlockingRecv error: recv got EOF from 127.0.0.1:57456 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 18:46:33.192428 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.191421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.18.235.189:42645)
0927 18:46:33.191527 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.191532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.191672 (+ 140us) server_negotiation.cc:407] Connection header received
0927 18:46:33.191806 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.191808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.191847 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.191916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.192335 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57472: BlockingRecv error: recv got EOF from 127.0.0.1:57472 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 18:46:33.227077 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.226173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.18.235.189:42645)
0927 18:46:33.226282 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.226286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.226420 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:46:33.226528 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.226531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.226574 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.226646 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.226990 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57476: BlockingRecv error: recv got EOF from 127.0.0.1:57476 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 18:46:33.293849 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.292921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.18.235.189:42645)
0927 18:46:33.293047 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.293051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.293175 (+ 124us) server_negotiation.cc:407] Connection header received
0927 18:46:33.293293 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.293296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.293344 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.293420 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.293755 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57480: BlockingRecv error: recv got EOF from 127.0.0.1:57480 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 18:46:33.424721 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.423754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.18.235.189:42645)
0927 18:46:33.423903 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.423907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.424017 (+ 110us) server_negotiation.cc:407] Connection header received
0927 18:46:33.424135 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.424137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.424184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.424262 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.424624 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57496: BlockingRecv error: recv got EOF from 127.0.0.1:57496 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 18:46:33.683655 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:33.682639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.18.235.189:42645)
0927 18:46:33.682749 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:33.682754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:33.682928 (+ 174us) server_negotiation.cc:407] Connection header received
0927 18:46:33.683061 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:33.683063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:33.683112 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:33.683170 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:33.683561 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57502: BlockingRecv error: recv got EOF from 127.0.0.1:57502 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:46:33.960466 9381 catalog_manager.cc:1259] Loaded cluster ID: 6de3773553434891a05bafad8e6088eb
I20260927 18:46:33.960538 9381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: loading cluster ID for follower catalog manager: success
I20260927 18:46:33.962296 9381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: acquiring CA information for follower catalog manager: success
I20260927 18:46:33.963392 9381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39e57e6f0acf4c62b8f4b101d8c1cfad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:46:33.987994 9395 catalog_manager.cc:1259] Loaded cluster ID: 6de3773553434891a05bafad8e6088eb
I20260927 18:46:33.988047 9395 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: loading cluster ID for follower catalog manager: success
I20260927 18:46:33.989667 9395 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: acquiring CA information for follower catalog manager: success
I20260927 18:46:33.990685 9395 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b9f492c836047c5bf21855e905b5beb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:46:34.201256 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:34.197519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.18.235.189:42645)
0927 18:46:34.197793 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:34.197797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:34.197815 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:46:34.197926 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:34.197928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:34.198054 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:34.198167 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:34.198370 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.198968 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.199877 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.200107 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.200191 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:34.200438 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:34.200874 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:34.200970 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:34.201031 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 18:46:34.201083 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260927 18:46:34.207419 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:34.204239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.18.235.188:35989)
0927 18:46:34.204440 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:34.204443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:34.204472 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:46:34.204545 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:34.204547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:34.204651 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:34.204733 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:34.204929 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.205458 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.206218 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.206427 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.206490 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:34.206712 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:34.207097 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:34.207178 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:34.207234 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 18:46:34.207280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260927 18:46:34.214293 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:34.210217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.18.235.188:35989)
0927 18:46:34.210305 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:34.210309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:34.210647 (+ 338us) server_negotiation.cc:407] Connection header received
0927 18:46:34.210800 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:34.210803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:34.210851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:34.210921 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:34.211213 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.211984 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.213145 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.213460 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.213554 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:34.213868 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:34.213943 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:34.214017 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:34.214108 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:46:34.214162 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 18:46:34.214359 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:34.210486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.18.235.189:42645)
0927 18:46:34.210596 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:34.210601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:34.210614 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:34.210659 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:34.210662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:34.210711 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:34.210810 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:34.211178 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.211984 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.213240 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.213475 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.213588 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:34.213874 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:34.213943 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:34.213992 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:34.214171 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 18:46:34.214223 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:34.214664 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:34.210149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.18.235.190:40565)
0927 18:46:34.210901 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:34.210905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:34.210917 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:46:34.211000 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:34.211003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:34.211142 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:34.211242 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:34.211534 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.212099 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.212990 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:34.213338 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:34.213446 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:34.213714 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:34.213966 (+ 252us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:34.214034 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:34.214269 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 18:46:34.214327 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":83,"threads_started":1}
I20260927 18:46:34.217480 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9146
I20260927 18:46:34.223061 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9221
I20260927 18:46:34.227679 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9295
2026-09-27T18:46:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2908 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:34 dist-test-slave-6lr6 krb5kdc[9512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:34 dist-test-slave-6lr6 krb5kdc[9512](info): set up 2 sockets
Sep 27 18:46:34 dist-test-slave-6lr6 krb5kdc[9512](info): commencing operation
krb5kdc: starting...
W20260927 18:46:35.173086 19374 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/test-user.keytab.
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
Entry for principal kudu/127.18.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab.
I20260927 18:46:35.278215 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35425
--webserver_interface=127.18.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35425,127.18.235.189:38343,127.18.235.188:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:35.379842 9525 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:46:35.380066 9525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:35.380087 9525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:35.380120 9525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:35.383904 9525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:35.383945 9525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:35.383962 9525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:35.383976 9525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:35.383992 9525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:35.388859 9525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35425,127.18.235.189:38343,127.18.235.188:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.190.keytab
--principal=kudu/127.18.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.190:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:35.389675 9525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:35.390513 9525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:35.398103 9533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:35.398123 9531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:35.398123 9530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:35.398263 9525 server_base.cc:1034] running on GCE node
I20260927 18:46:35.398948 9525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:35.399909 9525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:35.401065 9525 hybrid_clock.cc:648] HybridClock initialized: now 1790534795401052 us; error 25 us; skew 500 ppm
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:35.407436 9525 init.cc:377] Logged in from keytab as kudu/127.18.235.190@KRBTEST.COM (short username kudu)
I20260927 18:46:35.408216 9525 webserver.cc:460] Webserver started at http://127.18.235.190:45537/ using document root <none> and password file <none>
I20260927 18:46:35.408684 9525 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:35.408744 9525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:35.408919 9525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:35.410732 9525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data/instance:
uuid: "bc375ed0693a427fa7c1dfbc06158f7b"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "a9aafbb514c67acc957bcf03fc95988f"
server_key_iv: "601b69726dd9180b2a1b43e1827466d1"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.411252 9525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal/instance:
uuid: "bc375ed0693a427fa7c1dfbc06158f7b"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "a9aafbb514c67acc957bcf03fc95988f"
server_key_iv: "601b69726dd9180b2a1b43e1827466d1"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.414506 9525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:35.416592 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:35.417667 9525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:35.417768 9525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal
uuid: "bc375ed0693a427fa7c1dfbc06158f7b"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "a9aafbb514c67acc957bcf03fc95988f"
server_key_iv: "601b69726dd9180b2a1b43e1827466d1"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.417856 9525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:35.431593 9525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:35.434209 9525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:35.434360 9525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:35.440366 9525 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.190:35425
I20260927 18:46:35.440599 9592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.190:35425 every 8 connection(s)
I20260927 18:46:35.441206 9525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:35.442618 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9525
I20260927 18:46:35.442715 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-0/wal/instance
I20260927 18:46:35.442951 19374 external_mini_cluster.cc:1428] Setting key 8380d19f3eec50e6bf51e529d6bfb2a5
I20260927 18:46:35.443789 9593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:35.447324 9593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.189@KRBTEST.COM" created.
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:35.461025 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.449358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.18.235.190:35425)
0927 18:46:35.449785 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.449802 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.450011 (+ 209us) server_negotiation.cc:407] Connection header received
0927 18:46:35.450920 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.450925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.451100 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.451248 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.452051 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.452917 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.454270 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.454660 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.454842 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.456937 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.456943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.456951 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.456973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.458447 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.458921 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.458926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.458927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.459025 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.459329 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.459331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.459332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.459628 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.459718 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.460102 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.460289 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260927 18:46:35.461023 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.449358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35425 (local address 127.0.0.1:32936)
0927 18:46:35.449889 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.449923 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.450655 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.451258 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.451263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.451637 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.451886 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.451905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.453040 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.453044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.454146 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.454158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.454814 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.454819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.454917 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.455424 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.455440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.456800 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.458547 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.458556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.458565 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.458814 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.459105 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.459107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.459108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.459219 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.459769 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.459775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.459899 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.460035 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.460397 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":89,"spinlock_wait_cycles":45312,"thread_start_us":147,"threads_started":1}
I20260927 18:46:35.463169 9593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:35.463737 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.463546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38000)
0927 18:46:35.463630 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.463690 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:46:35.463922 9541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab.
Entry for principal kudu/127.18.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab.
I20260927 18:46:35.464624 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.464423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38008)
0927 18:46:35.464520 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.464569 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:46:35.464746 9593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:38343: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
I20260927 18:46:35.465167 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:38343
--webserver_interface=127.18.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab
--principal=kudu/127.18.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35425,127.18.235.189:38343,127.18.235.188:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:46:35.513983 9593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } attempt: 1
I20260927 18:46:35.514705 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.514430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38022)
0927 18:46:35.514566 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.514628 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:46:35.515326 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.515129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38024)
0927 18:46:35.515214 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.515262 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:46:35.515443 9593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:38343: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
W20260927 18:46:35.566028 9600 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:46:35.566252 9600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:35.566288 9600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:35.566354 9600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:35.570210 9600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:35.570255 9600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:35.570274 9600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:35.570310 9600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:35.570338 9600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:35.575222 9600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35425,127.18.235.189:38343,127.18.235.188:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.189.keytab
--principal=kudu/127.18.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.189:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:35.576074 9600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:35.576967 9600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:35.583580 9593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } attempt: 2
W20260927 18:46:35.584265 9608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:35.584439 9600 server_base.cc:1034] running on GCE node
I20260927 18:46:35.584477 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.584215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38038)
0927 18:46:35.584333 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.584388 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 18:46:35.584484 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:35.584627 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:35.585222 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.584991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38052)
0927 18:46:35.585088 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.585158 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 18:46:35.585238 9600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260927 18:46:35.585336 9593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.189:38343: Network error: Client connection negotiation failed: client connection to 127.18.235.189:38343: connect: Connection refused (error 111)
I20260927 18:46:35.586203 9600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:35.587357 9600 hybrid_clock.cc:648] HybridClock initialized: now 1790534795587341 us; error 27 us; skew 500 ppm
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:35.589789 9600 init.cc:377] Logged in from keytab as kudu/127.18.235.189@KRBTEST.COM (short username kudu)
I20260927 18:46:35.590533 9600 webserver.cc:460] Webserver started at http://127.18.235.189:33915/ using document root <none> and password file <none>
I20260927 18:46:35.591006 9600 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:35.591066 9600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:35.591230 9600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:35.593153 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data/instance:
uuid: "14c86f0374494607a9aa2b4cf8506d87"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "66c47e972bdadc105c18d73d88f6a86b"
server_key_iv: "5034d01fc44b521efe8e5834e4ca4660"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.593665 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal/instance:
uuid: "14c86f0374494607a9aa2b4cf8506d87"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "66c47e972bdadc105c18d73d88f6a86b"
server_key_iv: "5034d01fc44b521efe8e5834e4ca4660"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.596860 9600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:46:35.598986 9615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:35.600015 9600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:35.600117 9600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal
uuid: "14c86f0374494607a9aa2b4cf8506d87"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "66c47e972bdadc105c18d73d88f6a86b"
server_key_iv: "5034d01fc44b521efe8e5834e4ca4660"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.600198 9600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:35.612869 9600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:35.615526 9600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:35.615670 9600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:35.621702 9600 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.189:38343
I20260927 18:46:35.621984 9667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.189:38343 every 8 connection(s)
I20260927 18:46:35.622498 9600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/data/info.pb
I20260927 18:46:35.624879 9668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:35.628124 9668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:35.629431 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9600
I20260927 18:46:35.629542 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-1/wal/instance
I20260927 18:46:35.629820 19374 external_mini_cluster.cc:1428] Setting key 4cee54bd01f0f63a7632fd17a2dc8241
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.235.188@KRBTEST.COM" created.
I20260927 18:46:35.640344 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.629708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.18.235.190:35425)
0927 18:46:35.629834 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.629838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.630663 (+ 825us) server_negotiation.cc:407] Connection header received
0927 18:46:35.631657 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.631661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.631714 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.631793 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.632784 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.633613 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.634656 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.635015 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.635122 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.637187 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.637191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.637194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.637220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.638452 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.638948 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.638951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.638952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.639026 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.639398 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.639402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.639404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.639565 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.639646 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.640087 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.640197 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:35.640974 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.630077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35425 (local address 127.0.0.1:32952)
0927 18:46:35.630508 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.630558 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.631478 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.631925 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.631931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.632229 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.632648 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.632660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.633723 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.633726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.634539 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.634548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.635135 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.635138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.635260 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.635757 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.635771 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.637019 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.638554 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.638559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.638570 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.638859 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.639142 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.639144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.639145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.639261 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.639660 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.639666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.639869 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.640018 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.640280 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":84,"threads_started":1}
I20260927 18:46:35.642104 9668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:35.650452 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.642458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38068)
0927 18:46:35.642549 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.642561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.642618 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.643071 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.643074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.643226 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.643293 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.643298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.644151 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.644154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.644652 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.644659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.645174 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.645177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.645326 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.645700 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.645709 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.647676 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.649210 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.649213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.649214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.649399 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.649653 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.649656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.649658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.649745 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.650107 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.650110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.650191 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.650266 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.650347 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":46}
I20260927 18:46:35.650588 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.642552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.18.235.189:38343)
0927 18:46:35.642777 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.642790 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.642804 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:46:35.642840 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.642844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.642969 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.643073 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.643392 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.644053 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.644746 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.645068 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.645176 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.647786 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.647794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.647802 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.647828 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.649097 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.649484 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.649487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.649489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.649561 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.649828 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.649830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.649831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.650026 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.650134 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.650333 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.650438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab.
Entry for principal kudu/127.18.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab.
I20260927 18:46:35.651899 19374 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
/tmp/dist-test-taskJdx3KK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:44239
--webserver_interface=127.18.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab
--principal=kudu/127.18.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.235.148:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.235.190:35425,127.18.235.189:38343,127.18.235.188:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:46:35.651993 9668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:35.652544 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.652358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52012)
0927 18:46:35.652443 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.652499 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:46:35.652714 9619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
I20260927 18:46:35.653477 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.653274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52020)
0927 18:46:35.653374 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.653424 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:46:35.653627 9668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:44239: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
I20260927 18:46:35.676465 9593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } attempt: 3
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:35.684588 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.676806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38076)
0927 18:46:35.676969 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.676981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.677045 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.677260 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.677262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.677403 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.677798 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.677804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.678731 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.678733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.679241 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.679247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.680276 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.680278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.680357 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.680743 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.680753 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.681798 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.683394 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.683396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.683398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.683570 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.683828 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.683830 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.683831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.683893 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.684242 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.684244 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.684313 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.684384 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.684474 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 18:46:35.684675 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.676894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.18.235.189:38343)
0927 18:46:35.676995 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.677000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.677013 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:35.677125 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.677127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.677168 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.677246 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.677909 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.678641 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.679344 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.680184 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.680286 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.681948 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.681952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.681954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.681978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.683299 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.683649 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.683652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.683654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.683737 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.683969 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.683972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.683974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.684166 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.684258 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.684451 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.684553 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:46:35.685338 9593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:46:35.685956 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.685742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52028)
0927 18:46:35.685856 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.685900 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:46:35.686565 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.686354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52042)
0927 18:46:35.686478 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.686517 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:46:35.686667 9593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:44239: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
I20260927 18:46:35.702829 9668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } attempt: 1
I20260927 18:46:35.703320 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.703116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52044)
0927 18:46:35.703212 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.703262 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 18:46:35.703892 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.703698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52060)
0927 18:46:35.703785 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.703832 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 18:46:35.704001 9668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:44239: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
I20260927 18:46:35.717799 9593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } attempt: 1
I20260927 18:46:35.718276 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.718082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52074)
0927 18:46:35.718180 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.718222 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 18:46:35.718852 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.718691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52076)
0927 18:46:35.718765 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.718801 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:46:35.718962 9593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:44239: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
W20260927 18:46:35.755652 9674 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:46:35.755879 9674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:35.755916 9674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:35.755975 9674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:35.759835 9674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:35.759878 9674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:35.759896 9674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:35.759914 9674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:35.759940 9674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:35.764814 9674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.235.148:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.235.190:35425,127.18.235.189:38343,127.18.235.188:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/krb5kdc/kudu_127.18.235.188.keytab
--principal=kudu/127.18.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.235.188:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:35.765635 9674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:35.766479 9674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:35.772137 9668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } attempt: 2
I20260927 18:46:35.772727 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.772482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52078)
0927 18:46:35.772594 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.772650 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:46:35.773581 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.773340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52080)
0927 18:46:35.773453 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.773497 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:46:35.773743 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:35.773749 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:35.773874 9668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:44239: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
W20260927 18:46:35.774031 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:35.774266 9674 server_base.cc:1034] running on GCE node
I20260927 18:46:35.774717 9674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:35.775606 9674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:35.776767 9674 hybrid_clock.cc:648] HybridClock initialized: now 1790534795776739 us; error 41 us; skew 500 ppm
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:35.779186 9674 init.cc:377] Logged in from keytab as kudu/127.18.235.188@KRBTEST.COM (short username kudu)
I20260927 18:46:35.779891 9674 webserver.cc:460] Webserver started at http://127.18.235.188:35281/ using document root <none> and password file <none>
I20260927 18:46:35.780315 9674 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:35.780368 9674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:35.780517 9674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:35.782258 9674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data/instance:
uuid: "b063db8328b846c7962ac5aed9231a4b"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "b26d68b12bce124316a31278e7c58ece"
server_key_iv: "a8b1a8140fb6c93e294a9dfee469a08a"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.782702 9674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal/instance:
uuid: "b063db8328b846c7962ac5aed9231a4b"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "b26d68b12bce124316a31278e7c58ece"
server_key_iv: "a8b1a8140fb6c93e294a9dfee469a08a"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.785804 9674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:35.787820 9690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:35.788829 9674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:35.788919 9674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data,/tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal
uuid: "b063db8328b846c7962ac5aed9231a4b"
format_stamp: "Formatted at 2026-09-27 18:46:35 on dist-test-slave-6lr6"
server_key: "b26d68b12bce124316a31278e7c58ece"
server_key_iv: "a8b1a8140fb6c93e294a9dfee469a08a"
server_key_version: "encryptionkey@0"
I20260927 18:46:35.788992 9674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:35.794093 9593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } attempt: 2
I20260927 18:46:35.794678 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.794434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52094)
0927 18:46:35.794547 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.794597 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:46:35.795364 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.795166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52110)
0927 18:46:35.795260 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.795303 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 18:46:35.795488 9593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.235.188:44239: Network error: Client connection negotiation failed: client connection to 127.18.235.188:44239: connect: Connection refused (error 111)
I20260927 18:46:35.833724 9674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:35.836417 9674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:35.836588 9674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:35.842598 9674 rpc_server.cc:307] RPC server started. Bound to: 127.18.235.188:44239
I20260927 18:46:35.842831 9742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.235.188:44239 every 8 connection(s)
I20260927 18:46:35.843420 9674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/data/info.pb
I20260927 18:46:35.845810 9743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:35.846848 19374 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu as pid 9674
I20260927 18:46:35.846987 19374 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJdx3KK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534532155427-19374-0/minicluster-data/master-2/wal/instance
I20260927 18:46:35.847271 19374 external_mini_cluster.cc:1428] Setting key 9847429b01e438693c893852cdefa4e4
I20260927 18:46:35.849581 9743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:35.858848 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.850988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.18.235.190:35425)
0927 18:46:35.851102 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.851106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.851381 (+ 275us) server_negotiation.cc:407] Connection header received
0927 18:46:35.851504 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.851507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.851555 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.851638 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.852189 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.852660 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.853540 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.853745 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.853831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.856066 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.856069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.856070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.856092 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.857275 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.857670 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.857673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.857674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.857745 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.858004 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.858007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.858009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.858174 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.858244 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.858604 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.858710 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:46:35.859699 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.850939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.190:35425 (local address 127.0.0.1:32954)
0927 18:46:35.851265 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.851308 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.851425 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.851682 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.851686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.851966 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.852075 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.852081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.852763 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.852766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.853438 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.853447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.853865 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.853869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.853957 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.854359 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.854373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.855945 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.857366 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.857374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.857382 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.857581 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.857823 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.857826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.857828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.857924 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.858264 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.858270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.858393 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.858552 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.858824 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260927 18:46:35.859699 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.848368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.18.235.188:44239)
0927 18:46:35.848724 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.848737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.848769 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:46:35.849417 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.849433 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.849734 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.850034 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.850781 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.851974 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.852795 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.853351 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.853494 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.855677 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.855684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.855694 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.855714 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.857263 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.857643 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.857648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.857650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.857751 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.857996 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.857999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.858001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.858327 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.858512 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.858686 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.858982 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":227,"server-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260927 18:46:35.860788 9743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:35.865010 9668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } attempt: 3
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.190@KRBTEST.COM
I20260927 18:46:35.868328 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.861177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.189:38343 (local address 127.0.0.1:38090)
0927 18:46:35.861268 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.861279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.861344 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.861569 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.861572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.861718 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.861803 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.861808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.862727 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.862730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.863216 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.863223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.863631 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.863635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.863724 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.864086 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.864096 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.865201 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.866886 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.866889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.866891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.867109 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.867391 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.867394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.867396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.867469 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.867880 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.867883 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.867991 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.868075 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.868185 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":22}
I20260927 18:46:35.868700 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.861257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.18.235.189:38343)
0927 18:46:35.861373 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.861378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.861391 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:35.861431 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.861434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.861476 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.861541 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.862169 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.862631 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.863314 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.863540 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.863622 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.865352 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.865356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.865359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.865382 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.866760 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.867214 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.867217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.867219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.867293 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.867551 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.867554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.867556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.867736 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.867799 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.868146 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.868240 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:35.869278 9743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.189@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:46:35.870270 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.862096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.18.235.190:35425)
0927 18:46:35.862190 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.862194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.862206 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:46:35.862301 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.862304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.862340 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.862398 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.862901 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.863341 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.864142 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.864355 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.864435 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.867082 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.867085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.867088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.867111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.868532 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.868973 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.868977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.868978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.869055 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.869372 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.869375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.869377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.869570 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.869766 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.870042 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.870135 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:46:35.873215 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.865342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52138)
0927 18:46:35.865440 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.865452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.865515 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.865712 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.865715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.865889 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.865955 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.865960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.866844 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.866847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.867579 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.867588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.868156 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.868160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.868235 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.868612 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.868626 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.869943 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.871818 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.871822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.871824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.872047 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.872328 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.872331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.872333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.872421 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.872791 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.872794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.872877 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.872967 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.873070 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 18:46:35.873430 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.865394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.18.235.188:44239)
0927 18:46:35.865490 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.865494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.865506 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:46:35.865594 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.865597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.865632 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.865690 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.866049 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.866756 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.867687 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.868018 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.868439 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.870262 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.870266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.870268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.870288 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.871702 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.872149 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.872153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.872154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.872232 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.872515 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.872518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.872520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.872695 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.872785 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.873216 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.873310 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.188@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:46:35.875794 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: Bootstrap starting.
I20260927 18:46:35.878023 9668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:35.878646 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.869642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52148)
0927 18:46:35.870101 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.870113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.870168 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.871113 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.871116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.871622 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.871704 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.871709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.872471 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.872475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.873172 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.873182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.874152 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.874154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.874201 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.874540 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.874554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.875605 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.877127 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.877130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.877132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.877346 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.877621 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.877624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.877626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.877712 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.878161 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.878164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.878255 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.878359 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.878478 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399}
I20260927 18:46:35.878764 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.870167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.18.235.188:44239)
0927 18:46:35.870443 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.870451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.870464 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:35.870698 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.870701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.870745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.870823 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.871801 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.872372 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.873631 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.873994 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.874068 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.875724 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.875727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.875728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.875748 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.877005 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.877452 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.877455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.877457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.877539 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.877807 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.877830 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.877832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.878027 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.878114 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.878430 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.878537 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260927 18:46:35.878868 9668 log.cc:824] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.235.189@KRBTEST.COM
I20260927 18:46:35.881207 9743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: Bootstrap starting.
I20260927 18:46:35.881533 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: No bootstrap required, opened a new log
I20260927 18:46:35.882983 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.871404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.18.235.189:38343)
0927 18:46:35.871488 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.871492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.871516 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:46:35.871611 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.871614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.871648 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.871709 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.872357 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.872870 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.877179 (+ 4309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.877391 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.877479 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.879902 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.879905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.879908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.879930 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.881429 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.881860 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.881864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.881866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.881950 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.882207 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.882210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.882212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.882375 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.882526 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.882736 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.882841 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:46:35.883493 9743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:35.884239 9743 log.cc:824] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:35.885447 9668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.885665 9668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c86f0374494607a9aa2b4cf8506d87, State: Initialized, Role: FOLLOWER
I20260927 18:46:35.885903 19374 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:35.885973 9668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.886162 9743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: No bootstrap required, opened a new log
I20260927 18:46:35.887180 9668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:35.886974 9567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc375ed0693a427fa7c1dfbc06158f7b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970
I20260927 18:46:35.889940 9743 raft_consensus.cc:348] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.890134 9743 raft_consensus.cc:729] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b063db8328b846c7962ac5aed9231a4b, State: Initialized, Role: FOLLOWER
I20260927 18:46:35.890393 9743 consensus_queue.cc:260] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.890827 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.891037 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:35.891510 9743 sys_catalog.cc:564] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:46:35.891803 9762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:35.891866 9762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:35.893721 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.893848 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:35.890489 9600 master_runner.cc:186] Error getting master registration for 127.18.235.190:35425: OK, instance_id { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" instance_seqno: 1790534795438761 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:46:35.894097 9600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:46:35.894626 9593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } attempt: 3
W20260927 18:46:35.896361 9775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:35.896440 9775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:35.894533 9674 master_runner.cc:186] Error getting master registration for 127.18.235.190:35425: OK, instance_id { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" instance_seqno: 1790534795438761 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:46:35.897974 9674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 18:46:35 dist-test-slave-6lr6 krb5kdc[9512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.235.190@KRBTEST.COM for kudu/127.18.235.188@KRBTEST.COM
I20260927 18:46:35.902832 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.895364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.18.235.188:44239)
0927 18:46:35.895467 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.895471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.895487 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:46:35.895525 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.895528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.895571 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.895655 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.896049 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.896743 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.897768 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.898084 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.898165 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.900205 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.900208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.900211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.900233 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.901448 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.901815 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.901819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.901821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.901886 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.902115 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.902119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.902120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.902266 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.902344 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.902590 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.902689 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:46:35.902832 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.894957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.235.188:44239 (local address 127.0.0.1:52154)
0927 18:46:35.895161 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:35.895173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:35.895235 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:35.895694 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:35.895697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:35.895861 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:35.895948 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.895954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.896857 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.896861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.897640 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:35.897652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.898192 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.898197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:35.898283 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.898714 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:35.898730 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:35.900096 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:35.901543 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.901546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.901548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.901729 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.901963 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:35.901966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:35.901968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:35.902041 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:35.902352 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:35.902355 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:35.902440 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 18:46:35.902571 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:46:35.902690 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260927 18:46:35.904966 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: Bootstrap starting.
I20260927 18:46:35.906792 9593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:35.907395 9593 log.cc:824] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:35.909236 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: No bootstrap required, opened a new log
I20260927 18:46:35.911892 9593 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.912072 9593 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc375ed0693a427fa7c1dfbc06158f7b, State: Initialized, Role: FOLLOWER
I20260927 18:46:35.912328 9593 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.912693 9778 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.912820 9778 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:35.913359 9593 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:35.913465 9567 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:46:35.913615 9567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:35.914772 9567 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.916306 9642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14c86f0374494607a9aa2b4cf8506d87"
I20260927 18:46:35.917055 9642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:46:35.917922 9789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:35.917981 9789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:35.918335 9642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc375ed0693a427fa7c1dfbc06158f7b in term 1.
I20260927 18:46:35.918766 9541 leader_election.cc:302] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c86f0374494607a9aa2b4cf8506d87, bc375ed0693a427fa7c1dfbc06158f7b; no voters:
I20260927 18:46:35.919026 9778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:35.919221 9778 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [term 1 LEADER]: Becoming Leader. State: Replica: bc375ed0693a427fa7c1dfbc06158f7b, State: Running, Role: LEADER
I20260927 18:46:35.919503 9778 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } }
I20260927 18:46:35.919718 9717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b063db8328b846c7962ac5aed9231a4b"
I20260927 18:46:35.919932 9717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:35.921044 9791 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc375ed0693a427fa7c1dfbc06158f7b. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.921171 9791 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:35.921578 9717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc375ed0693a427fa7c1dfbc06158f7b in term 1.
I20260927 18:46:35.922405 9793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:35.922626 9567 leader_election.cc:288] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [CANDIDATE]: Term 1 election: Requested vote from peers 14c86f0374494607a9aa2b4cf8506d87 (127.18.235.189:38343), b063db8328b846c7962ac5aed9231a4b (127.18.235.188:44239)
I20260927 18:46:35.923208 9793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:35.923744 9525 master_runner.cc:263] No leader master found from master bc375ed0693a427fa7c1dfbc06158f7b
I20260927 18:46:35.923799 9525 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:46:35.928656 9717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [term 1 FOLLOWER]: Refusing update from remote peer bc375ed0693a427fa7c1dfbc06158f7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:35.929344 9791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:35.929561 9642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [term 1 FOLLOWER]: Refusing update from remote peer bc375ed0693a427fa7c1dfbc06158f7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:35.930122 9791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:35.932956 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc375ed0693a427fa7c1dfbc06158f7b. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.933074 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:35.936094 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc375ed0693a427fa7c1dfbc06158f7b. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.936213 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:46:35.936671 9546 tablet.cc:2369] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:35.943240 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.925392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.18.235.188:44239)
0927 18:46:35.925486 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.925490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.933363 (+ 7873us) server_negotiation.cc:407] Connection header received
0927 18:46:35.933546 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.933549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.933597 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.933650 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.934651 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.935351 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.936492 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.936836 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.936919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.939672 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.939676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.939679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.939708 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.941353 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.941845 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.941849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.941851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.941939 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.942272 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.942277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.942279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.942497 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.942605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.942969 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.943078 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:46:35.945962 9791 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.946077 9791 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:35.946116 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.925305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.18.235.190:35425)
0927 18:46:35.925409 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.925414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.926171 (+ 757us) server_negotiation.cc:407] Connection header received
0927 18:46:35.926357 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.926361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.926407 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.926459 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.927467 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.928173 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.938057 (+ 9884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.938362 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.938444 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.940054 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.940059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.940062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.940089 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.941622 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.943798 (+ 2176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.943802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.943804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.943898 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.945142 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.945146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.945148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.945358 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.945459 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.945861 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.945972 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:46:35.946197 9798 mvcc.cc:205] Tried to move back new op lower bound from 7334030524114288640 to 7334030524089339904. Current Snapshot: MvccSnapshot[applied={T|T < 7334030524114288640}]
I20260927 18:46:35.947127 9751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.947261 9751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:35.948987 9804 mvcc.cc:205] Tried to move back new op lower bound from 7334030524114288640 to 7334030524089339904. Current Snapshot: MvccSnapshot[applied={T|T < 7334030524114288640}]
I20260927 18:46:35.950886 9778 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.951037 9778 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:35.952121 9802 mvcc.cc:205] Tried to move back new op lower bound from 7334030524114288640 to 7334030524089339904. Current Snapshot: MvccSnapshot[applied={T|T < 7334030524114288640}]
I20260927 18:46:35.952291 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc375ed0693a427fa7c1dfbc06158f7b" member_type: VOTER last_known_addr { host: "127.18.235.190" port: 35425 } } peers { permanent_uuid: "14c86f0374494607a9aa2b4cf8506d87" member_type: VOTER last_known_addr { host: "127.18.235.189" port: 38343 } } peers { permanent_uuid: "b063db8328b846c7962ac5aed9231a4b" member_type: VOTER last_known_addr { host: "127.18.235.188" port: 44239 } } }
I20260927 18:46:35.952402 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:46:35.953449 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:35.925827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.18.235.189:38343)
0927 18:46:35.925933 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:35.925938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:35.940821 (+ 14883us) server_negotiation.cc:407] Connection header received
0927 18:46:35.941746 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:35.941784 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:35.941839 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:35.941914 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:35.943001 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.943659 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.947670 (+ 4011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:35.947997 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:35.948078 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:35.949864 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:35.949869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:35.949871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:35.949898 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:35.951512 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.952040 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.952043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.952045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.952137 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:35.952475 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:35.952479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:35.952481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:35.952692 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:35.952819 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:35.953194 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 18:46:35.953303 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:35.954614 9793 catalog_manager.cc:1347] Generated new cluster ID: ed4b89a445df47ca9fc84ad77eb9395c
I20260927 18:46:35.954679 9793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:35.972635 9793 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:35.973816 9793 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:35.986403 9793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc375ed0693a427fa7c1dfbc06158f7b: Generated new TSK 0
I20260927 18:46:35.987143 9793 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:46:36.014173 9748 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.011994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.18.235.188:44239)
0927 18:46:36.012100 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.012105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.013232 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 18:46:36.013414 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.013418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.013461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.013507 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.014059 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52176: BlockingRecv error: recv got EOF from 127.0.0.1:52176 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 18:46:36.015434 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.012658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.18.235.190:35425)
0927 18:46:36.012752 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.012756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.014604 (+ 1848us) server_negotiation.cc:407] Connection header received
0927 18:46:36.014809 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.014812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.014853 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.014898 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.015323 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33002: BlockingRecv error: recv got EOF from 127.0.0.1:33002 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 18:46:36.017094 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.015960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.18.235.189:38343)
0927 18:46:36.016053 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.016058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.016309 (+ 251us) server_negotiation.cc:407] Connection header received
0927 18:46:36.016465 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.016468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.016525 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.016572 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.016997 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38116: BlockingRecv error: recv got EOF from 127.0.0.1:38116 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 18:46:36.020733 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.019466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.18.235.189:38343)
0927 18:46:36.019551 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.019554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.019911 (+ 357us) server_negotiation.cc:407] Connection header received
0927 18:46:36.020095 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.020099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.020159 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.020205 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.020653 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38126: BlockingRecv error: recv got EOF from 127.0.0.1:38126 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 18:46:36.030964 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.029731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.18.235.189:38343)
0927 18:46:36.029816 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.029819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.030216 (+ 397us) server_negotiation.cc:407] Connection header received
0927 18:46:36.030386 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.030389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.030431 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.030475 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.030888 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38142: BlockingRecv error: recv got EOF from 127.0.0.1:38142 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 18:46:36.035979 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.034766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.18.235.189:38343)
0927 18:46:36.034849 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.034852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.035243 (+ 391us) server_negotiation.cc:407] Connection header received
0927 18:46:36.035409 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.035412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.035452 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.035496 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.035903 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38148: BlockingRecv error: recv got EOF from 127.0.0.1:38148 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 18:46:36.043043 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.041814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.18.235.189:38343)
0927 18:46:36.041921 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.041925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.042314 (+ 389us) server_negotiation.cc:407] Connection header received
0927 18:46:36.042482 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.042485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.042525 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.042567 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.042969 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38156: BlockingRecv error: recv got EOF from 127.0.0.1:38156 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 18:46:36.054138 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.052835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.18.235.189:38343)
0927 18:46:36.052922 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.052926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.053372 (+ 446us) server_negotiation.cc:407] Connection header received
0927 18:46:36.053541 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.053544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.053585 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.053631 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.054063 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38160: BlockingRecv error: recv got EOF from 127.0.0.1:38160 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 18:46:36.074023 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.072401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.18.235.189:38343)
0927 18:46:36.072513 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.072517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.073018 (+ 501us) server_negotiation.cc:407] Connection header received
0927 18:46:36.073266 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.073270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.073324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.073391 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.073919 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38174: BlockingRecv error: recv got EOF from 127.0.0.1:38174 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 18:46:36.109274 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.108126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.18.235.189:38343)
0927 18:46:36.108237 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.108242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.108483 (+ 241us) server_negotiation.cc:407] Connection header received
0927 18:46:36.108608 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.108610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.108656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.108719 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.109164 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38190: BlockingRecv error: recv got EOF from 127.0.0.1:38190 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 18:46:36.176252 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.175178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.18.235.189:38343)
0927 18:46:36.175307 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.175312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.175442 (+ 130us) server_negotiation.cc:407] Connection header received
0927 18:46:36.175583 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.175585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.175632 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.175698 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.176140 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38206: BlockingRecv error: recv got EOF from 127.0.0.1:38206 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:46:36.307387 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.306077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.18.235.189:38343)
0927 18:46:36.306211 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.306216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.306399 (+ 183us) server_negotiation.cc:407] Connection header received
0927 18:46:36.306611 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.306615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.306676 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.306762 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.307266 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38216: BlockingRecv error: recv got EOF from 127.0.0.1:38216 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 18:46:36.566310 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:36.565267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.18.235.189:38343)
0927 18:46:36.565414 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:36.565419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:36.565484 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:46:36.565633 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:36.565636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:36.565684 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:36.565750 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:36.566216 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38224: BlockingRecv error: recv got EOF from 127.0.0.1:38224 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:46:36.893144 9762 catalog_manager.cc:1259] Loaded cluster ID: ed4b89a445df47ca9fc84ad77eb9395c
I20260927 18:46:36.893218 9762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: loading cluster ID for follower catalog manager: success
I20260927 18:46:36.894860 9762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: acquiring CA information for follower catalog manager: success
I20260927 18:46:36.895961 9762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14c86f0374494607a9aa2b4cf8506d87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:46:36.897420 9775 catalog_manager.cc:1259] Loaded cluster ID: ed4b89a445df47ca9fc84ad77eb9395c
I20260927 18:46:36.897475 9775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: loading cluster ID for follower catalog manager: success
I20260927 18:46:36.898887 9775 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: acquiring CA information for follower catalog manager: success
I20260927 18:46:36.899820 9775 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b063db8328b846c7962ac5aed9231a4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:46:37.083635 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:37.080140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.18.235.189:38343)
0927 18:46:37.080376 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:37.080379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:37.080394 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:37.080529 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:37.080532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:37.080649 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:37.080743 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:37.080972 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.081496 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.082295 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.082514 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.082584 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:37.082921 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:37.083334 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:37.083420 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:37.083445 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 18:46:37.083489 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260927 18:46:37.089468 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:37.086261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.18.235.188:44239)
0927 18:46:37.086481 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:37.086484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:37.086498 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:46:37.086595 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:37.086597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:37.086697 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:37.086782 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:37.086921 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.087411 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.088178 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.088437 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.088504 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:37.088726 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:37.089070 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:37.089200 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:37.089226 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 18:46:37.089271 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260927 18:46:37.095793 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:37.092192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.18.235.189:38343)
0927 18:46:37.092323 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:37.092327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:37.092559 (+ 232us) server_negotiation.cc:407] Connection header received
0927 18:46:37.092650 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:37.092653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:37.092717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:37.092795 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:37.093026 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.093716 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.094834 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.095135 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.095210 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:37.095412 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:37.095465 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:37.095527 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:37.095629 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:46:37.095676 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:46:37.096423 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:37.092298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.18.235.188:44239)
0927 18:46:37.092411 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:37.092415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:37.093120 (+ 705us) server_negotiation.cc:407] Connection header received
0927 18:46:37.093248 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:37.093251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:37.093294 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:37.093362 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:37.093558 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.094213 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.095135 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.095433 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.095511 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:37.095778 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:37.095849 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:37.095911 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:37.096250 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 18:46:37.096300 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:46:37.096480 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:37.091956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.18.235.190:35425)
0927 18:46:37.092152 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:37.092156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:37.092245 (+ 89us) server_negotiation.cc:407] Connection header received
0927 18:46:37.092403 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:37.092406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:37.092556 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:37.092648 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:37.092835 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.093552 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.094405 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:37.094749 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:37.094835 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:37.095646 (+ 811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:37.095949 (+ 303us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:37.096022 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:37.096282 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 18:46:37.096337 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260927 18:46:37.098938 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9525
I20260927 18:46:37.104100 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9600
I20260927 18:46:37.109050 19374 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJdx3KK/build/debug/bin/kudu with pid 9674
2026-09-27T18:46:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2881 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5789 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (264947 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST